Google's Open Knowledge Format is quite a bombshell standard.
No RAG, no nothing, just a wiki in a git repo is enough to make LLMs work!
Some progressive disclosure by very basic dir/file naming tricks and index.md
files. Some minimal key-value metadata.
That is great, but the key limitation is still the LLM context window. The stream of LLM attention has to be directed to useful and consistent work. Some say the "smart zone" is under 250K tokens, others prefer 200K or even 50K. It can only do this much while staying in that zone. A smaller project can fit in the context window, sure. What if the project is big and ongoing?
As this knowledge accumulates, it rots. An LLM-created wiki is particularly prone to that. As things unfold, some decisions get course-corrected. Some code was just slop, but the LLM took it as an example for future work. Some random claims from random sources get into the wiki. Overall quality of LLM texts is not good. The "knowledge base" accumulates contradicting, stale, or false claims. An LLM is free to pick a side randomly there. So, stale or false info gets smeared over the wiki and the codebase. In the end, you no longer know what is in there and you no longer want to read it.
The ability to course correct is exactly the ability to absorb feedback. Feedback leads to design and policy changes, code changes, bugs fixed, features added. A good product is a result of that steady process. Good architecture lets you accumulate more feedback without going into full entropy mode.
My current project structure is a stack of data tiers, which mostly live in git repos.
Google OKF would only cover some tiers here. Basically, only the things I can put into one LLM-accessible wiki: (3)-(4). (1)-(2) may also go there, with some risk. I pack (5) with (1),(2),(3) now, but that is already a stretch. Volume, lifespans and the general genre are all different. Unlike the wikis, the ticket corpus is supposed to grow indefinitely. So is the chat archive. Adding 20-30 tickets a day is nothing special. Chat logs, same, and no one wants to read them all. As we go (1) to (7), the volume of data grows by a factor at every step. This cannot be the same repo. Not even two. Maybe three: wiki, code, and tickets/logs/facts. Four is even better and seven is even more so. That is, assuming you have one repo for the project's code. I have more, at least six: the system, the runtime, tests and key library dependencies.
To maintain coherence between all of those, I employ permalinks. Those differ from wiki links in three key aspects:
The syntax for permalinks is much more relaxed, e.g. index/index.js:24
is valid.
Just a path and a line and maybe a commit hashlet, index/index.js:24:Mw
.
The hashlet helps to locate the line in any later version of the file,
be it shifted, re-edited or reformatted.
That really saves context: we pick the right snippets cheaply and in bulk.
To automate permalink use, bee
has two commands: cite
and cited
.
cite
shows a file interleaved with snippets it links to.
cited
shows it interleaved with snippets that link to it. That way I solve both retrieval and coherence. The latter becomes a much bigger task as the project grows. Checking the state of inbound/outbound links trivially exposes staleness.
We arrange texts, code and artifacts along the "idea⟶reality" axis. The mission of information architecture is to make information "easy to find, navigate, and understand". I can start by authoring the seed documents and then keep building into outer tiers. I build, I face difficulties, I fix and course-correct.
This architecture absorbs feedback, but does not devolve into slop.