How the hell are AI APIs still basically stateless?
It can't be true that the whole context needs to be sent every single time. Even if we use prefix caching or a durable object it not really a solution!
And yes, some startups are building stuff for retrieval . But they are all . mostly semantic search on md files. nothing changed in years on the retrieval part except a small layer that adds timestamps.
That is not a persistent context. That is us carrying the model’s brain around in freaking md files.
I was building an agent that did direct calls to the biggest the best model i could find on cloudflare (kimi k3), but i wanted to test if a stupid chromium page can do a better job. Instead of calling cloudflare, i asked it to open a page, go to z.ai and use it as the intelligence layer for the complex tasks, and because z.ai context window remains consistence across different calls, it performed better than kimi k3! The z.ai page wasn't even logged.
It can't be that a local session is able to maintain context for 30+ minutes but an API call can't maintain context for more than 8 minutes. This is really stupid.
i am writing a paper about agentic memory, but i never thought it would be any good...
then i went & read most adjacent papers that is written around that topic, some of which from openai employees. almost all of them were awful! i never expected to see such quality coming from the supposable top labs.
wait! "quality" is not the right word to describe this... maybe "groupthink" is the word. Every paper is circling around the same abstract more or less, almost every single one is build around the same worldview %10 better retrieval !
i think we fear openai & anthropic way too much, they are starting to stagnate the same as all big companies. if i put myself in the shoes of someone who works there, it would be impossible to go against what everyone else is thinking.
god, i am half way across the planet & its still very hard to fight against silicon valley Ai groupthink.
Comments URL: [https://news.ycombinator.com/item?id=49424697](https://news.ycombinator.com/item?id=49424697)
Points: 1