Building an AI-Native Knowledge Base with Apache Jena

A developer describes building an AI-native knowledge base using Apache Jena, leveraging LLMs to generate RDF triples and query a Fuseki server, aiming to create a specialized reasoning layer that supplements general-purpose AI.

You know what the LLM wiki is missing? It can ingest, lint, and query — but what about reason ? Wouldn’t it be nice if your knowledge base mimicked your brain a little closer? We humans learn incrementally, build upon previously learnt concepts, and update our priors in light of new data. What if we could do that with AI — build a second brain with specialized reasoning? AGI is too big a target for a solo developer like me, and I’m not even interested in it. What has helped me throughout my career is not general knowledge but specialized skills that make me valuable. Specialized AI supplements the general-purpose AI we are lucky to have at our disposal today. One doesn’t replace the other; it just tries to fix the last-mile connectivity issue I find with AI today: it gets you tantalizingly close to the solution, then falls short and goes haywire. Apache Jena https://jena.apache.org/ — and for that matter the whole community of ontology modellers — has been working patiently all these years. AI has made it clear that this is their moment to shine. Ontologies, knowledge graphs, SHACL: terms now bandied about everywhere have made people aware of this formerly niche corner of software engineering. I want to share my take and my approach, using a stack I have been familiar with for many years. I had dabbled with RDF, OWL and SWRL using Protégé https://protege.stanford.edu/ before AI. I found it a useful mental exercise to rethink concepts under that framework, but ultimately it remained a hobby project. I could never get my team excited about it, because I knew that for it to work, it needs scale. I’ve long wanted to translate my entire domain knowledge at work into OWL ontologies, but that is an endeavour that demands quality work, and the ROI was hard to justify before. AI changed the game. It seems to me that LLMs have a very fundamental familiarity with the language of RDF graphs. A few months ago, I moved to a new team internally after many years in my prior one. Over those years I had built up mental models in my head that made me very good at my job. After experiencing fatherhood, I wanted a fresh start and a new challenge, so I requested a move. But I knew I did not want to spend years of organic learning to catch up with the new team. Instead, I started using AI to build myself an LLM wiki. My idea was that, going forward, the AI I work with needs to know what I know, so that I can delegate tasks to it with more confidence. And I had MCP servers at my disposal to access pretty much every source of data I needed at my job. The LLM wiki looked promising at first, but I ran into my first issue: AI slop. I was producing textbooks’ worth of notes and reading none of it. So I tried asking the LLM to produce notes in the form of RDF triples instead, my reasoning being that a triple is the TikTok equivalent of a paragraph in this attention-starved world we cohabit. To my surprise, I got back a perfectly syntactic Turtle file. That led me down a rabbit hole. I knew about Apache Jena from a pre-AI-era exploration into rule engines, so I ran a Fuseki https://jena.apache.org/documentation/fuseki2/ server locally, asked Claude Code to ingest into it and query from it — and it just knew what to do. I’ve long suspected that when LLMs started offering reasoning, and personalities for configuring that reasoning, the language matched that of the ontologies I was familiar with. In some ways it felt like having stumbled upon the source code of LLMs, because to date I am yet to encounter even a syntactic issue. Soon I was well on my way to building my personal knowledge base, using it like a memory layer by adding hooks — run ingest when I find something, run query on every request… crude commands, but they worked. With syntax off the table, the next issue I encountered was consistency. In one session, a class gets created for Service. In the next, the same concept is modelled as Component. I was manually renaming things at this point, because I was more interested in getting to a sizeable knowledge base quickly. But I knew I was running into the same problem in a new costume: the feeling of productivity, but unusable slop. I did try alternatives like the simpler SKOS https://www.w3.org/TR/skos-reference/ framework: everything is a Concept that belongs to a ConceptScheme and can be related as narrower, broader, or related. Simple enough to constrain the LLM, good enough to start with — but too broad and general-purpose for my use case. So I finally decided to build my own knowledge graph system, with the goal of capturing a specialized domain of knowledge that I can deploy to assist with my office tasks, and eventually some team tasks like on-call triaging and assistance. A modest goal. Apache Jena has been around almost since the first papers in this space. It is not the most well-known or exciting Apache project, but a glance at the Maven repository ranks jena-core as the 1 in RDF Libraries and is used as dependency in many other projects. That's understandable — ontologies have always been around, and they are very useful at a low level. But Jena has a very steep learning curve; documentation exists, but it is of an obtuse nature that takes time to untangle. There just hasn't been enough reason for it to become more high-level and user-friendly. Until now. Jena has ready-made, production-grade offerings for every component you may need to build a knowledge graph — all free, completely open source: LLMs could generate .ttl files just fine, but there was no cohesion. So I decided to build an MCP server as the sole means of interaction with the knowledge graph API. In this server, I expose tools to add a single class, a single object property, a single data property, a single individual. This lets me enforce maximum gatekeeping on what goes into the knowledge graph. Another set of tools is a mix of deterministic and general SPARQL queries. And a single skill, exposed as an MCP resource, instructs the LLM to first search for existing classes and properties before adding new ones — which solves the consistency problem. I’m not interested in over-engineering this. I know that over time I will add more features and it will bloat, but simplicity is key with AI. You don’t need to reinvent the wheel each time. Just as you don’t micromanage a human employee, you just need to be there to provide a gentle nudge or a bit of direction from time to time — and trust the employee, or the AI, to get the job done. I hope you learnt something new today. This is the first post in a series where I will go deeper into the system I have built — the stack, the code, and my designs — so do give me a follow. Building an AI-Native Knowledge Base with Apache Jena https://pub.towardsai.net/building-an-ai-native-knowledge-base-with-apache-jena-77fbefe206f1 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.