{"slug": "proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence", "title": "Proveniq: An AI License Review Is Only Useful When It Can Show Its Evidence", "summary": "An engineer built Proveniq, an open-source AI-assisted asset provenance and license-screening tool that uses Gemma to extract license details while keeping the model from inventing evidence or making final compliance decisions. In an evaluation of 28 license notices, Gemma extracted license identifiers for all 27 nonblank notices, attribution requirements for 21, and creator information for 12, with zero invented creator or attribution details and 15/15 documented rule-engine outcomes matched.", "body_md": "**What happens when you download an asset, use AI to review its license, and discover later that you cannot prove where the information came from?**\n\nImagine finding the perfect illustration, icon, font, or code snippet for a project. You save it, start building, and move on. A few weeks later, you need to answer some basic questions:\n\nSometimes, the information is scattered across webpages, README files, license notices, and personal notes. Sometimes the creator is unknown. Sometimes the license is ambiguous.\n\nAnd sometimes, an AI confidently gives you an answer that sounds correct but has no reliable evidence behind it.\n\n**That last problem is what I wanted to address.**\n\nI built **Proveniq**, an open-source, AI-assisted asset provenance and license-screening tool designed to help developers organize asset information, inspect license terms, and identify issues that need human review.\n\nThe idea is simple: **AI can help interpret license information, but it should not be allowed to invent evidence or make the final decision.**\n\nProveniq lets you record an asset's name, source URL, creator information, license details, and intended use in one place.\n\nIt then helps turn the available license text into structured information, connects extracted details to supporting quotations, and checks supported license conditions against the intended use.\n\nThe goal is not to replace legal review. It is to make missing information, potential conflicts, and uncertainty easier to see before they become problems.\n\nA typical workflow looks like this:\n\nThe important part is what happens when the system *cannot* establish an answer.\n\nIt should expose the uncertainty—not hide it behind a polished AI response.\n\nMy first priority was deciding what the language model should **not** control.\n\nGemma is useful for extracting candidate information from unstructured text. But extracting a license identifier is not the same as proving that an asset can legally be used in a particular situation.\n\nThose are different responsibilities.\n\nIn Proveniq:\n\nThis separation matters because a model can produce valid-looking structured output while still getting the underlying facts wrong.\n\nA JSON response is not proof. A plausible license name is not proof. Even a correct license identifier does not automatically establish that a specific use complies with every applicable condition.\n\nI wanted the system to make those distinctions visible.\n\nThe stack is intentionally straightforward:\n\nThe model works with the license text supplied to the application. The system validates the structure of its response and checks supporting quotations rather than blindly trusting the generated fields.\n\nI also worked on failure handling and evidence integrity.\n\nFor example, when the source license text or source URL changes, stale extraction results are cleared. When relevant evidence or the intended use changes, an earlier assessment can become outdated. Quotations are checked against the supplied text, and unsupported cases are not treated as successful reviews.\n\nThe application also handles inference failures without pretending an extraction succeeded.\n\nThese details are less exciting than a flashy AI demo, but they are central to a tool whose output people may rely on when managing software and creative assets.\n\nI wanted to evaluate more than whether the application could return JSON.\n\nI created a versioned fixture set containing **28 license notices**. Of these, 27 were sent to Gemma; the blank notice was rejected with an HTTP 422 response.\n\nHere are the results from that evaluation:\n\n| Measurement | Result | \n|---|---|\n| Nonblank notices evaluated by Gemma | 27 | \n| License identifiers extracted | 27/27 | \n| Attribution requirements extracted | 21/27 | \n| Creator information extracted | 12/27 | \n| Invented creator or attribution details | 0 | \n| Documented rule-engine outcomes matched | 15/15 | \n\nThe results highlight an important distinction.\n\nThe model extracted a license identifier from every nonblank fixture, but attribution and creator information were not always available. Those missing fields are not something an application should silently fill in.\n\nThe fixture suite also covers unsupported or ambiguous cases—including MIT/WTFPL cases, prompt-injection attempts, draft studio terms, and ambiguous license tokens—that should lead to an unknown result or manual review rather than an invented clearance.\n\nThe rule-engine result measures agreement with the documented outcomes in that test set. It does **not** establish universal legal correctness.\n\nThe automated suite currently passes **118 backend tests and 19 frontend tests**. Type checking, linting, and the production build also passed in the recorded checks.\n\nThere are still limits to what these results establish: the final automated suite uses mocked inference rather than a live Ollama evaluation, and a successful build is not the same as a verified production deployment.\n\nI would rather state those limitations clearly than make the project look more complete than it is.\n\nThe most important lesson was that reliability cannot come from prompting alone.\n\nA better prompt can improve extraction, but it cannot guarantee that every answer is correct. The application needs checks outside the model: schema validation, evidence verification, explicit handling of unsupported cases, and rules that can be tested independently.\n\nI also learned that missing data deserves first-class treatment.\n\nIf Proveniq cannot establish the creator, it should say the creator is unknown. If it cannot support a license interpretation, it should request review. If the source changes, the old assessment should not quietly remain authoritative.\n\nIn other words:\n\n**The absence of evidence is a result the application must preserve, not a gap for AI to fill.**\n\nThat principle influenced the product more than any individual prompt.\n\nLicense and provenance information can be part of a developer's workflow every day. I wanted to explore whether a small open-weight model could make that workflow more organized without turning the model into an unquestioned authority.\n\nUsing Gemma through Ollama gives the project an inspectable, replaceable inference layer rather than making the core workflow dependent on a closed, metered model API.\n\nThe deterministic checks remain ordinary application code. The data model, validation logic, and evaluation fixtures can be inspected and improved independently of the model.\n\nThat separation also makes the project easier to challenge. Developers can question a rule, add a fixture, reproduce a failure, or improve extraction without needing to accept a black-box verdict.\n\nOpen source does not automatically make an AI system reliable. But it makes the system's assumptions and limitations easier to examine—and gives other developers a way to improve them.\n\nI am building it for developers, indie makers, and small teams who reuse code or creative assets and want a more disciplined way to track provenance and review license information.\n\nA team could use it as an early screening step before incorporating an asset into a project. It could also help create a record of what was known, which evidence supported a decision, and which questions remained unanswered.\n\nIt is not a substitute for legal advice, and a generated report is not a guarantee that an asset is legally cleared. The value is in organizing the evidence and surfacing uncertainty earlier.\n\nMy next priorities are to:\n\nI would especially like feedback on the cases where a license-screening tool should stop automatically and ask a human to decide.\n\nThe live demo and repository links should be filled in before publishing. I don't want to present a deployment as live until it has been verified.\n\nThese are the challenge categories I would consider for Proveniq, subject to the official rules and verification that the final submission qualifies.\n\nThese are potential category entries, not a claim that the project has already qualified for every track. I will list only the categories supported by the final, verified submission.\n\nAI is very good at making incomplete information look complete.\n\nFor a license-screening tool, that is exactly the behavior I want to guard against.\n\nProveniq is my attempt to build a more careful alternative: use AI to help extract information, preserve the evidence behind it, let deterministic code handle supported rules, and make uncertainty visible when the system cannot justify a conclusion.\n\nIt is still a work in progress, and the evaluation is deliberately limited. But I think that is a worthwhile place to start.\n\n**If you build software with third-party assets, what would you need to see before trusting an AI-assisted license review?**\n\nI'd love feedback on the evidence model, the edge cases, and where the tool should draw the line between automation and human review.", "url": "https://wpnews.pro/news/proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence", "canonical_source": "https://dev.to/priyanshupatel/proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence-112f", "published_at": "2026-10-19 05:47:26+00:00", "updated_at": "2026-10-19 05:48:04.916813+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-safety", "artificial-intelligence"], "entities": ["Proveniq", "Gemma"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence", "markdown": "https://wpnews.pro/news/proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence.md", "text": "https://wpnews.pro/news/proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence.txt", "jsonld": "https://wpnews.pro/news/proveniq-an-ai-license-review-is-only-useful-when-it-can-show-its-evidence.jsonld"}}