Flagship models are not always better

A developer building a realtime note-taking app found that Apple's on-device Foundation Models failed to process a 27-minute conversation, while a smaller Qwen model succeeded using a hierarchical map-reduce technique. The benchmark reveals that flagship models are not always better for long-context tasks.

Frontier Models Suck Sometimes A Benchmark Written on Jul 8, 2026 • 18 minute read • Share on 𝕏 https://x.com/intent/tweet?url=I%20just%20read%20this%20insightful%20blog%20post%20by%20%40tejask%20at%20https%3A%2F%2Ftej.as%2Fblog%2Fnote-app-ai-benchmark-apple-intelligence-sucks I’ve been building a small realtime note taking app. It’s like one button: you tap it, you talk on your iPhone or from your wrist on the Apple Watch, and it hands your rambling thought back to you as text plus like a summary, some action items you buried, and some questions you left dangling. On-device speech-to-text, then “AI analysis,” then explore. The word carrying the entire app on its back in that sentence is AI . The “AI” that was counting words I decided to start small: the first version’s “AI” was just a heuristic, algorithm-based solution not generative . It counted words. The “topics” it proudly surfaced were, literally, just the most frequent tokens in your transcript. So it would inform you, with total confidence, that today’s themes were “know” and “it’s” . I looked at it and was like bruh this isn’t AI, it’s Array.prototype.sort .At this point I was like okay, maybe I don’t need regex for this but like a real language model. But it had to be on-device and private. It also needed to process loooooooooong conversations, like 2-hour podcast episodes and things. How can we process that when language models often come with smaller context windows https://en.wikipedia.org/wiki/Context window ? Hierarchical map-reduce A language model can only look at so much text at once: a budget called its context window . Apple’s on-device model tops out around 4,000 tokens where a token is roughly 1 english word . Even a small Qwen model that could fit on an iPhone I downloaded it isn’t much roomier: a 27-minute ramble is many times that. So you run straight into a genuinely strange question: how do you summarize something that doesn’t even fit inside the reader’s head? Picture summarizing a fat book when you can only hold a few pages in your mind at a time. You wouldn’t try to swallow it whole, you’d read one chunk, jot the key points on an index card, and move on. Chunk, card. Chunk, card. Do that for the whole book and you’ve quietly turned one impossible task into a stack of very possible ones. In other words, Map : slice the transcript into bite-sized pieces that do fit, and run the model on each piece on its own. Each pass pulls out just that slice’s key points, action items, and questions. It’s called “map” because you apply the same little function across every piece. This is the same concept as. array.map Reduce : now gather all those index cards into a pile and write the final answer from the cards , not from the original book. Merge the duplicates, drop the overlap, fold it all into one clean set plus a title and a summary. “Reduce” just means folding many things down into one. This is the same concept as. array.reduce Hierarchical : when there are so many chapters that even the pile of index cards won’t fit on the table, group the cards, summarize each group onto a new card, and do it again: summaries of summaries until the whole thing finally fits. Like folding a long letter in half, and in half again, until it’s pocket-sized. A short note never needs that extra fold, a genuinely multi-hour recording does, and this is what keeps the whole thing from tipping over. Apple Intelligence said no The tasteful, on-brand move for a privacy-first notes app is Apple’s Foundation Models https://developer.apple.com/documentation/foundationmodels . Real LLMs on the device without talking to the cloud that just work offline. Wow. I wired it in behind the analyzer, kept my chunked map-reduce pipeline, and fed it a real recording: a genuinely private, 27-minute conversation with my wife https://de.linkedin.com/in/lea-artemov-839107239/ about random stuff. Half an hour of half-formed thoughts about moving countries, a birthday, the price of a Tesla in Berlin, the people at last week’s meetings. The exact kind of sprawling human talk this app exists to make sense of. Apple’s model looked at it and returned: Detected content likely to be unsafe. At this point, even though there was nothing unsafe in it, I turned the safety dial all the way down to Apple’s most permissive setting permissiveContentTransformations https://developer.apple.com/documentation/foundationmodels/systemlanguagemodel/guardrails/permissivecontenttransformations and it still refused. Apple Intelligence is such garbage. I just want to transcribe a voice note on device from a regular conversation and it refuses. Absolute trash. — Tejas Kumar @TejasKumar View tweet A guardrail that fires on a private note about a birthday isn’t a guardrail but instead some bullshit that ruins people’s experience. I needed something better. So I downloaded a brain Pivot: open weights. Qwen3-4B-Instruct-2507 https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF , Apache-2.0, quantized to a 4-bit GGUF. This weighs pun intended in at ~2.3 GB and runs fully on-device through llama.rn https://github.com/mybigday/llama.rn . Same hierarchical map-reduce pipeline, but I just swapped the model using react-native-ai https://www.react-native-ai.dev/ . No cloud, no key, no guardrail. It analyzes whatever you record, because it’s yours and it never leaves. It works. Getting 2.3 GB of model to actually run on an iPhone was two good battles. Battle one: memory On the CPU it was so slow the app looked hung to the point where you start checking the debugger to see if you’ve deadlocked yourself. Move it to the GPU n gpu layers: 99 , full Metal offload and it flies, but mapping ~2.5 GB into resident memory gets you instantly jetsammed on an 8 GB iPhone. The fix is a single entitlement, com.apple.developer.kernel.increased-memory-limit https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.increased-memory-limit . This btw is Apple quietly conceding that your phone can, in fact, hold other language models in addition to its own. 🤫 Battle two: the model thought itself to death Qwen3 is a “thinking” model. Its chat template opens a <think block and reasons before answering. Delightful, except my token budget is finite and it spent the entire thing thinking and never got around to emitting the JSON. Literally like some people, “all thinking no doing” . Every note came back empty. The one-line fix was to turn off the thinking: chat template kwargs: { enable thinking: false; } Then it started to work However while the little 4B on my phone produced genuinely structured insight, private and free and offline, the data was really not usable: it understood the conversation , but produced pretty trashy summaries with even worse action items and questions. This was more noise than signal from my conversation. Eventually, I decided I need a bit more power: it’s time to consider maybe possibly using cloud models. The Benchmark In 2026, is this the best we can do? Or am I shipping a cute compromise and calling it a feature? Locally, the ceiling unfortunately was the Qwen model I tried for an iPhone 16 Pro Max : it had high scores in benchmarks and looked SOTA State of the Art for the iPhone level. This is of course did not include cloud models. In an effort to find the best of the best without the hardware constraint, I did a benchmark. Here’s how I did it. Methodology The data. One input, held constant: the transcript of that same 27-minute voice note. Every model saw the identical text. I’m benchmarking the analysis , not the transcription, so the speech-to-text step is upstream and the same for everyone. Whatever imperfections the transcript had and it had some, it’s a real recording , everyone shared them. The candidates. 11 models: - My on-device Qwen3-4B, plus 10 cloud models via OpenRouter https://openrouter.ai , Anthropic https://anthropic.com/ ’s Claude Sonnet 5- Opus 4.8, OpenAI https://openai.com/ ’s GPT-5.5 Pro, Google https://google.com/ ’s Gemini 3.1 Pro, xAI https://x.ai/ ’s Grok 4.3, DeepSeek https://deepseek.com/ ’s V4 Pro,- Qwen3-Max-Thinking, - Kimi K2.6, z-ai https://z-ai.com/ ’s GLM-4.7, and- GLM-5.2. The task. Every model got the exact same prompt, asking for structured JSON so the outputs were directly comparable: You analyze a spoken voice-note transcript into structured insights. Return ONLY a JSON object with EXACTLY these keys: {"title": string <=8 words, no quotes/date , "summary": string 3-5 sentences , "actionItems": string , "questions": string , "keyPoints": string , "topics": string } - keyPoints/actionItems/questions: quote the speaker's wording, high-signal only. - topics: 1-3 word subject tags. Use ONLY what is in the transcript. No text outside the JSON. The on-device model can’t hold the whole transcript at once the context window problem from earlier , so it does the same job through the map-reduce, with two smaller prompts. The “map” prompt runs on every chunk: You extract structured insights from ONE segment of a longer spoken voice note. Use ONLY what is in this segment. Do not invent people, dates, or facts. - keyPoints: the few most important ideas or decisions, each self-contained. - actionItems: concrete tasks someone intends to do imperative . - questions: open or unresolved questions raised. - topics: short 1-3 word subject tags. Quote the speaker's actual wording for key phrases; do not paraphrase. Prefer fewer, higher-signal items. and the “reduce” prompt folds the chunk cards into one title and summary: Write a specific title <= 8 words and a 3-5 sentence executive summary for a voice note, using ONLY the provided key points. Neutral and factual. The scoring. I anonymized all 10 outputs my phone was blind “Candidate A”, nothing more and scored them with a panel of three LLM judges, each given a different personality and each scoring blind on a six-axis rubric summary accuracy, summary quality, action items, questions, key points, topics , zero to ten: Judge 1: a ruthless magazine editor who despises filler, vague summaries, and padded lists. Judge 2: an executive chief-of-staff who only values accurate, actionable takeaways and hates noise. Judge 3: a faithfulness auditor who hunts hallucinations and any claim not grounded in the transcript. Three lenses, blind, so nobody could reward a model for its logo. The final rank is the average of the three overall scores. The fine print, so you can rerun it. Every call ran at temperature 0.2, one pass each, in July 2026, through OpenRouter, and every candidate was scored by the identical three-judge panel and rubric. Holding the judge configuration fixed across everything you compare is the one rule you cannot break if the numbers are going to mean anything ref: An Empirical Study of LLM-as-a-Judge https://arxiv.org/pdf/2506.13639 . Prompts, models, and settings are all up above since reporting the exact model, prompt, and parameters is what separates a benchmark from a vibe ref: reproducibility is a cornerstone of the method https://www.ncbi.nlm.nih.gov/books/NBK547531/ . Results Here’s what shook out: | Rank | Model | Overall | Latency | |---|---|---|---| | 🥇 | Kimi K2.6 | 7.23 | 16.9s | | 🥈 | GPT-5.5 Pro | 7.07 | 361.2s 🥱 | | 🥉 | GLM-4.7 | 6.67 | 3.4s | | 4 | GLM-5.2 | 6.67 | 43.7s | | 5 | Claude Opus 4.8 | 6.50 | 14.0s | | 6 | DeepSeek V4 Pro | 6.50 | 27.2s | | 7 | Gemini 3.1 Pro | 6.37 | 25.5s | | 8 | Claude Sonnet 5 | 5.77 | 11.7s | | 9 | Qwen3-Max-Thinking | 5.53 | 8.3s | | 10 | Grok 4.3 | 5.37 | 13.3s | | 11 | Qwen3-4B my phone | 5.10 | local | What I learned A few things genuinely surprised me. A “cheap” model beat the whole frontier. Kimi K2.6 took it, sweeping all three judges editor 7, chief-of-staff 7, auditor 7.7 . A clear +0.16 over GPT-5.5 Pro and +0.56 over the GLMs. It was the only candidate that did the actual hard part of the job: twenty-seven minutes of two people rambling contains exactly one real to-do, buried in an aside “I owe people slides and stuff.” Kimi found precisely that, wrote it down, and invented nothing else. Everyone else either dumped raw quotes or padded the list with things that were never tasks. Kimi compressed, the rest re-transcribed. The frontier flagships were expensive and underwhelming. GPT-5.5 Pro scored a strong second on the rubric 7.07 and then took 361 seconds ~6 whole minutes to do it For a one-button phone app where you tap and expect an answer, six minutes is literally a bug. This is, to me, unshippable. It also sits in the priciest tier while it makes you wait. My beloved Claude got lazy. Opus 4.8 landed 5th and Sonnet 5 landed 8th. Not from hallucinating btw : both were honest and accurate. They lost because they quote-dumped instead of synthesizing. Sonnet’s “questions” list was, verbatim, conversational filler like "Are you recording now?" and "Did you have any problems today?" . It also missed the one real action item entirely. The whole field basically tied on the summary : the entire ranking was decided by whether a model synthesized the structured fields or just regurgitated. Almost everyone regurgitated. Newer and bigger was not better. More on that in a second, because it’s the most interesting result in the table. And my phone came dead last, unsurprisingly. 5.10 to Kimi’s 7.23. This is the best I can do with a 4B model on my phone. The GLM-5.2 question: is the new flagship worth it? This was the comparison I most wanted, and the answer was refreshingly blunt: no. GLM-5.2 is z-ai https://z-ai.com/ ’s newest and largest 1M context window model, the flagship, the one you’d instinctively reach for over its cheaper sibling. On this task it tied GLM-4.7 to the decimal : 6.67 overall, with identical scores from all three judges 6.5 / 6.4 / 7.1 . Same quality. For that tie it charged you ~ 13x the latency 43.7s vs 3.4s and a higher price tier. Worse, where it differed, it differed down on exactly the fields that decide this benchmark. GLM-4.7 listed one action item: the real one, “I owe people slides.” GLM-5.2 kept that and bolted on “I think you just tell them I don’t know what’s happening” … which is not a task, it’s a fragment of the conversation. Its questions got mushier too, sliding back toward the same "Are you recording now?" filler the Claudes lost on. On the judges’ numbers: action items dropped from 4.5/5/5.5 to 3.5/4/4, questions from 4/3.5/5.5 to 2.5/3.2/5. It bought a hair of summary polish by regressing on the load-bearing structured extraction, but lost the structured fields where it mattered most. GLM-4.7 dominates GLM-5.2 for this use case: same quality, ~13x faster, cheaper, better action items. The newer model wasn’t an upgrade, it was a wash that regressed where it counted. That surprised me more than anything else in the run, and it’s a useful reminder that “flagship” is a marketing tier, not a benchmark result. About that last place Here’s the part that really stings because the entire promise of this app is trust: my on-device model didn’t just lose on sparseness, it made shit up. Twice. Two of the three judges independently caught it inverting the core fact of the whole conversation. I was talking about how I live in Germany and would maybe consider moving to America. Specifically, I said “I hate the system, but I love the land,” about America , weighing a German passport as the safety net that finally makes the move possible because Germany’s a great fallback I have a complicated healthcare situation /blog/health-fitness-devrel-hemophilia . My model confidently wrote it backwards, as if America is home and Germany is the destination It flipped an extremely important fact of the recording Then it invented a task out of thin air: Check if the recording of the app is available and accessible within 12 minutes. There is no such thing. It stitched that “12-minute” to-do together from someone offhandedly mentioning that ground beef needs twelve minutes to cook. Amusingly, Qwen3’s much larger cloud sibling, Qwen3-Max-Thinking, flipped the exact same geography. A family trait, apparently and a hint that the fabrication is baked into the lineage, not the quantization. For a notes app, a hallucinated action item is worse than a missing one. Sparse is a bug, confident and wrong is a betrayal. If my app puts a fake to-do on your list, you’ve lost the one thing a notes app is for . And yet the 4B on my phone 5.10 out-scored two frontier cloud models on the rubric: Grok 4.3 5.37, and it’s only 0.27 ahead and, on structure, held its own against Qwen3-Max-Thinking 5.53 . A 4-billion-parameter model running on the GPU in my pocket, no network, no bill, landing within striking distance of models a thousand times its size. That is genuinely, unreasonably impressive. It’s just not yet trustworthy . What this benchmark can’t tell you I want to be straight about where this is a fun blog experiment and not a real paper, because a good study is loud about its own limits ref: Ten Simple Rules for Writing Research Papers https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003453 . Read the decimals as vibes, not physics. n = 1. This is one transcript, from one conversation, in one domain me rambling with my wife . It shows how these models handled this note, not yours. Not justifying your sample size is, ironically, the exact hole most LLM evaluations fall into ref: empirical study of LLM judges https://arxiv.org/pdf/2506.13639 . One run, no error bars. Models sample and so do the judges, but I ran each exactly once. So I can’t show you the variance, and a rigorous version would run it many times and report the spread. Unforunately I didn’t want to spend waste? a lot of money running it many times. The judges are LLMs, not humans. LLM-as-judge is genuinely useful, but it carries known biases, and high agreement between judges does not prove they are measuring the same thing a human would ref: reliability without validity https://arxiv.org/pdf/2606.19544 . Three prompted personas blunt the logo bias, they do not turn the panel into ground truth. I spoke about this recently at AI Engineer World’s Fair https://ai.engineer/worldsfair/2026 in San Francisco . Shared transcript, shared flaws. The imperfect speech-to-text was identical for everyone, which is fair, but it means a slice of what I measured is “who copes best with messy transcription,” not pure understanding. None of this dents the headline a cheap model synthesized where the frontier regurgitated, and the little on-device one confidently made things up . But if you want to act on this, run it on your own recordings before you trust the ranking. So, is the AI on my phone good enough? Nope. Well, at least not yet. Not on its own, and certainly not for the trust of a user. The bigger lesson, the one I didn’t expect to write: the frontier is not where the interesting movement is. The best output in my test came from a model most people haven’t heard of. The best value ran in 3.4 seconds for pennies. The most expensive, most-hyped flagships either timed themselves out of contention or quietly regressed against their own cheaper siblings. The gap from my phone to the frontier is about 2.1 points and it’s closing fast enough that I’d bet the next 4B closes most of it. So where the app actually lands I flipped my own plan I was all set to ship on-device by default and treat the cloud as a scary little opt-in. Then the benchmark happened and embarrassed me out of that position. The 4B on my phone totally made shit up and flipped the geography of my own life, inventing a to-do out of ground beef in the process. The trustworthy output, the one that found the single real action item and invented nothing, came from a cloud model. For a notes app, trust is the entire product, and the trustworthy answer was in the cloud. So I flipped it. Cloud is the default now. On-device is the offline fallback. That said, I refuse to hand some random inference vendor a transcript of me and my wife talking about moving countries, my healthcare /blog/health-fitness-devrel-hemophilia , and our friends with my name stapled to the top. So everything from here down is how exactly I plan to get the same/similar benefits of on-device processing but in the cloud. The plan is cloud by default, but built so the vendor learns as little about you as I can manage. Some text has to reach the model to get analyzed though. There’s really no way around that. It’s the same reason Apple can’t end-to-end-encrypt iCloud Mail: somebody, somewhere, has to read the cleartext ref: Advanced Data Protection https://support.apple.com/guide/security/advanced-data-protection-for-icloud-sec973254c5f/web . So we can’t do “send nothing”, but we can shrink what the model sees down to almost nothing, making it really hard to tell that nothing is you . Here are three ways I plan to pull that off: 1. Misdirection, so the vendor can never guess who you are A vendor can tie a request back to a real human through four independent channels: - your network identity the source IP , - your account a login or API key , - your payment trail, and - the PII personally identifiable information sitting inside the transcript itself. They’re four separate locks so you need a separate key for each. The whole design principle is that no single party in the pipeline ever holds two channels it can link together ref: Cloudflare https://blog.cloudflare.com/stronger-than-a-promise-proving-oblivious-http-privacy-properties/ . Defense in depth: break one lock and you’ve still got nothing. The good news is those four locks aren’t equally hard. The scariest one for a notes app, the PII sitting right there in the transcript text, also happens to be the cheapest to kill. So that’s the one I’m building first. The IP and the account channels need a lot more machinery, so they come later. - Hide the IP with OHTTP Oblivious HTTP . OHTTP RFC 9458 https://www.rfc-editor.org/rfc/rfc9458.html splits a request across two parties that are not allowed to collude. The client encrypts the request with HPKE and posts the ciphertext to a relay , which blindly forwards it to a gateway , which decrypts it and calls the model. The relay sees your IP but only ciphertext; the gateway “does not learn the Client’s IP address or any other identifying information” and only ever sees the relay’s IP ref: RFC 9458 https://www.rfc-editor.org/rfc/rfc9458.html .One party knows who , the other knows what , and it stays that way “unless the relay and gateway collude” ref: Cloudflare https://blog.cloudflare.com/stronger-than-a-promise-proving-oblivious-http-privacy-properties/ , which is why they have to be run by different companies. This is exactly how Apple’s Private Cloud Compute https://security.apple.com/blog/private-cloud-compute/ routes requests “through an OHTTP relay, operated by a third party, which hides the device’s source IP address before the request ever reaches” the model. The plan is to run my own gateway behind a third-party relay, so the relay sees a phone talking gibberish and the gateway sees text arriving from some other server.The catch is that the whole trick only works if the relay and the gateway are run by different companies . If they’re both me, or both one big CDN, then the unlinkability is just theatre. So the real work here isn’t code, it’s finding a relay operator who isn’t also my gateway host and keeping that relationship alive. This is probably the heaviest lift on the list. - Decouple the account. There’s no per-user login with the vendor, and the model’s API key lives only on my gateway. So every request goes out under the same shared credential, and from the vendor’s side all my users blur into one caller. Who is that? No idea. Could be anyone. The gap right now is that the credential is a single shared secret baked into the app, so the next step is minting a short-lived per-device token on the server, so one leaked secret can’t impersonate everybody at once. Apple does the fancy version with single-use blind-signature credentials that “authorize valid requests without tying them to a specific user” ref: Private Cloud Compute https://security.apple.com/blog/private-cloud-compute/ . I’d like to copy that. - Scrub the PII Kinda table stakes and the biggest bang for buck: remove all the personally identifiable information before it leaves the device. Like, think about what’s actually in a transcript: names, employers, phone numbers, all of it. But iOS already ships on-device named-entity recognition in the before it ever leaves the phone. Natural Language framework https://developer.apple.com/documentation/naturallanguage . You can infer people, places, organizations, fully offline. I can reach it from React Native with a small native module. No model to download, no network call. On top of that I use the Microsoft Presidio https://github.com/microsoft/presidio reversible-pseudonymization trick: “Peter” becomes <PERSON 1 , every single time, and the model only ever sees the placeholders. The map from <PERSON 1 back to “Peter” never leaves the device, so only your phone can turn the answer back into real names. This is the layer that saves you even if every other layer fails. A totally compromised vendor sitting on raw plaintext just sees <PERSON 1 owes <ORG 2 some slides , and has no clue who that is. Now, is it perfect? Nope. On-device recognition will miss a name here and there, or misread a weird one. So I keep it conservative, keep the map on the phone, and treat it as one layer of defense, not a magic eraser. If the built-in recognizer isn’t good enough, tiny redactor models like OpenAI’s open-weight Privacy Filter https://openai.com/index/introducing-openai-privacy-filter/ are a later upgrade . - Cover the boring channels too. These are mostly cheap and mostly soon. Batching and padding requests on a slightly random schedule, and stripping identifying headers, are all app-level stuff that rides along with the gateway work anyway OHTTP leaves timing and size correlation out of scope, so mild batching is the sensible patch . Encrypted Client Hello https://blog.cloudflare.com/announcing-encrypted-client-hello/ is a CDN toggle, not app code. A boring, standard TLS fingerprint I get for free just by using the phone’s normal networking stack, so I blend into the crowd. 2. Encrypt everything the app stores Everything the app stores audio, transcript, the insights, even the titles and timestamps should get encrypted on the device before it ever touches disk, so my own backend only ever holds ciphertext. The interesting question is where the key comes from. To begin, I make a random 256-bit key per note, seal each payload with XChaCha20-Poly1305 there are solid, maintained crypto libraries for React Native, or I lean on SQLCipher down at the database layer , and wrap the master key with a Secure Enclave https://developer.apple.com/documentation/security/protecting-keys-with-the-secure-enclave key that never leaves the chip and unlocks with Face ID. That alone gets me encrypted-at-rest, hardware-backed, on this device. It also sits on a data model that Proton https://proton.me/blog/proton-pass-security-model and Standard Notes https://standardnotes.com/help/security/encryption have already battle-tested: one random key per note , each wrapped by the master key. I encrypt the metadata too titles, tags, timestamps , because Signal’s whole sealed sender https://signal.org/blog/sealed-sender/ lesson is that locking the letter but printing the address on the outside is only half a job. Eventually we can go multi-device. The WebAuthn PRF extension https://www.corbado.com/blog/passkeys-prf-webauthn lets a passkey cough up a deterministic 32 bytes of secret from a Face ID gesture, which I run through HKDF into a key. The neat part is that a passkey synced through iCloud Keychain carries that secret to every device you own, so the same key just shows up on your iPad and MacBook and your notes decrypt themselves. No key server, no me in the middle. I love it. It’s also the newest, least-trodden path on this whole list, so it waits until the basic version is rock solid. Since a passkey that fails to sync would take your notes down with it, I’d like to ship it with a printable recovery key so losing every device isn’t the same as losing your notes. The end state is basically Apple’s Advanced Data Protection: keys live only on your trusted devices, recovery is a code you hold, and the provider me simply can’t read your stuff ref: ADP https://support.apple.com/guide/security/advanced-data-protection-for-icloud-sec973254c5f/web . 3. Don’t just trust the vendor, verify it can’t read you My winning model, Kimi K2.6, is open-weight , and in 2026 open-weight models can run inside a GPU confidential-computing enclave that you can actually verify. Phala https://phala.com/confidential-ai-models lists Kimi K2.6 on Intel TDX plus NVIDIA H100, behind an OpenAI-compatible endpoint. The model runs in hardware-encrypted memory that’s sealed even from the machine’s own operator, and the enclave hands back a signed attestation quote proving exactly which code and which weights are about to touch your transcript ref: NEAR AI https://docs.near.ai/cloud/private-inference/ , Tinfoil https://tinfoil.sh/technology . TLS terminates inside the box, not at some load balancer that could peek. The old complaint about all this was speed, and that’s basically gone now: overhead is down around 95 to 99% of native https://www.spheron.network/blog/confidential-gpu-computing-nvidia-tee-encrypted-vram/ . This is probably gonna be a longer effort, but initially for the MVP I plan to start with OpenRouter’s one-click Zero Data Retention https://openrouter.ai/docs/features/privacy-and-logging , routed only to endpoints that don’t store or train on your data, prompt logging off. It’s a config flag and really more of a promise than a proof, but it’s a decent start imo. The provider still sees plaintext while it’s working, and you’re trusting them to keep the contract. But it can ship today, and paired with the on-device PII scrubbing from move 1, what they see is already <PERSON 1 owes <ORG 2 slides . The latter part, the actual enclave, leans on stuff I don’t control. First, it only exists if a vendor is genuinely hosting the model I want, in a TEE, at a price and speed I can live with. Second, parsing one of those attestation quotes on the phone is a real project, so the sane version verifies the quote on my gateway which I run and already trust instead of asking a React Native app to do enclave forensics. Change the base URL, check the quote there, and you’ve turned “trust the vendor’s word” into “verify it”. And per my own rule that the user should always see what the system’s doing, the app already tells you which engine ran each note: cloud Kimi, on-device Qwen, or the offline heuristic. The goal is to grow that little banner into a “verified enclave” badge once the attestation is wired up for real. Where the trust boundary actually sits So here’s where you actually land: the PII column and the shared-account column are the near-term wins, and the IP column is still waiting on that relay. There’s exactly one unavoidable moment where plaintext has to exist off your phone, and it’s the instant the model reads the already-redacted transcript to analyze it. No amount of crypto the cryptography kind, not the web3 scam kind makes that go away. But look at how little of “you” is left by the time that text shows up: | Party in the pipeline | Your IP? | Readable content? | Your account / payment? | Your name / PII? | |---|---|---|---|---| | OHTTP relay | Yes | No ciphertext | No | No | | My gateway | No relay’s IP | Yes, but de-PII’d | Shared key only | No | | The model vendor | No gateway’s IP | In an enclave, no; on ZDR, yes but not retained | Login-less, crypto-funded | No it’s <PERSON 1 | No single row holds two channels it can link. Subpoena the relay and you get an IP attached to gibberish. Breach the vendor and you get <PERSON 1 owes <ORG 2 slides , unstored, with no IP and no name. Pull the payment trail and you get a prepaid top-up attached to no usage. Some risks that still remain are that the relay and gateway could collude so they’re run by different orgs , timing and size correlation survives content encryption so, batching and padding , and my on-device scrubber could miss a name so I keep it conservative and keep the reversible map on-device . None of those give anyone the full picture. So yeah, that’s the plan. Maybe a bit overengineered for a passive AI notetaker for myself but I love the craft so who cares? If you’ve shipped a small model into a real product, watched one confidently invent your homework out of ground beef, or built the misdirection stack above and can tell me where I’m wrong, please let me know on 𝕏 https://x.com/tejaskumar .