{"slug": "how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded", "title": "How to Build Retrieval Audit Logs for Internal Engineering PDFs in Go (Grounded)", "summary": "A developer outlined a design for retrieval audit logs in Go that treat each RAG answer over internal engineering PDFs as an evidence ledger rather than a transcript, pinning corpus revisions, ordered chunk IDs, and selected citations per request. The approach withholds any answer whose citations cannot be joined to that request's retrieval set, and separates citation validity into its own SLO distinct from request availability. The author argues the invariant catches fabricated identifiers and stale cross-request citations but does not prove the prose accurately reflects the cited passage.", "body_md": "Short answer: build retrieval audit logs as an evidence ledger, not a transcript. For every answer over the PDF folder, pin the corpus revision, preserve the ordered chunk IDs returned by retrieval, record the citations selected by generation, and withhold any answer whose citations cannot be joined to that request's retrieval set.\n\nSuppose the page fires at 02:17. The on-call sees a user report that an internal documentation answer cites `release-runbook.pdf, page 18`, but the active PDF has only 16 pages. CPU is quiet, latency is inside its objective, and the generator returned fluent prose. None of those signals answers the operational question: which indexed revision and which passages were allowed to support this answer?\n\nThe earlier signal should have been a failed evidence join.\n\nRetrieval-augmented generation combines generated output with retrieved non-parametric memory, and the original RAG paper identifies provenance as a benefit of that arrangement. An application still has to turn provenance into an enforceable release rule. For this system, I would make citation validity a separate SLO from request availability — a response can be fast and syntactically valid while being unsuitable for release.\n\nStart with governance, before picking an index or writing an event struct. Decide who may inspect a question, who may inspect retrieved PDF passages, how long each class of record survives, and which fields must remain usable after a serving component is replaced. A useful audit event needs enough information to reproduce the evidence boundary without copying every sensitive question, answer, and PDF passage into a second data store. Record a request ID, authenticated actor or calling service, event time, corpus snapshot ID, query digest, retrieval-policy version, ordered chunk IDs and scores, prompt-template version, answer digest, cited chunk IDs, validation result, latency, and final disposition.\n\nNames are display data, not identity. `release-runbook.pdf` can be replaced while keeping the same filename, so derive a source revision from its content digest. A chunk identifier should also bind the source revision to a page range, chunking-policy version, and ordinal. That gives the evidence ledger stable handles across index rebuilds and makes a corpus snapshot a concrete manifest rather than a timestamp someone hopes is close enough.\n\nDon't log credentials, authorization headers, or unrestricted extracted text. Auditability does not justify widening access to internal engineering documentation. If retention policy prohibits raw questions and answers, their digests plus access-controlled references can still support correlation and replay checks; if investigators must inspect content, keep that content behind the same authorization boundary as the source PDFs.\n\nThe retention classes don't have to match. Operational metrics can expire quickly, evidence events can remain for the investigation window, and source content can follow the documentation system's own policy. What matters is preserving the join keys for as long as the organization claims an answer is auditable. Deleting a PDF passage while retaining an event that points to it may satisfy an event-retention rule, but it no longer permits substantive replay; that limitation should be explicit.\n\nThe key invariant is deliberately narrow: every citation on a released answer must identify a chunk retrieved for that same request under the pinned corpus snapshot. This catches fabricated identifiers, stale cross-request citations, and broken correlation. It does not prove that the prose accurately represents the cited passage. That requires a separate evaluation of support, which is why one green `citation_valid` field must never be presented as total answer quality.\n\nKeep that contract in application-owned fields rather than opaque component state. The following Go program hashes text that should not be copied into the event, checks citation membership and duplication, and emits one append-oriented JSON record. The sample data is illustrative; the invariant and portable identifiers are the parts to carry into production.\n\n```\npackage main\n\nimport (\n    \"crypto/sha256\"\n    \"encoding/hex\"\n    \"encoding/json\"\n    \"errors\"\n    \"fmt\"\n    \"os\"\n    \"time\"\n)\n\ntype RetrievedChunk struct {\n    ID    string  `json:\"id\"`\n    Score float64 `json:\"score\"`\n}\n\ntype GeneratedAnswer struct {\n    Text      string   `json:\"text\"`\n    Citations []string `json:\"citations\"`\n}\n\ntype AuditEvent struct {\n    RequestID        string           `json:\"request_id\"`\n    OccurredAt       time.Time        `json:\"occurred_at\"`\n    CorpusSnapshot   string           `json:\"corpus_snapshot\"`\n    RetrievalPolicy  string           `json:\"retrieval_policy\"`\n    QueryDigest      string           `json:\"query_digest\"`\n    Retrieved        []RetrievedChunk `json:\"retrieved\"`\n    Citations        []string         `json:\"citations\"`\n    AnswerDigest     string           `json:\"answer_digest\"`\n    CitationValid    bool             `json:\"citation_valid\"`\n    FinalDisposition string           `json:\"final_disposition\"`\n}\n\nfunc digest(value string) string {\n    sum := sha256.Sum256([]byte(value))\n    return hex.EncodeToString(sum[:])\n}\n\nfunc validateCitations(retrieved []RetrievedChunk, citations []string) error {\n    if len(citations) == 0 {\n        return errors.New(\"answer has no citations\")\n    }\n\n    allowed := make(map[string]struct{}, len(retrieved))\n    for _, chunk := range retrieved {\n        allowed[chunk.ID] = struct{}{}\n    }\n\n    seen := make(map[string]struct{}, len(citations))\n    for _, citation := range citations {\n        if _, ok := allowed[citation]; !ok {\n            return fmt.Errorf(\"citation %q was not retrieved\", citation)\n        }\n        if _, duplicate := seen[citation]; duplicate {\n            return fmt.Errorf(\"citation %q is duplicated\", citation)\n        }\n        seen[citation] = struct{}{}\n    }\n    return nil\n}\n\nfunc main() {\n    retrieved := []RetrievedChunk{\n        {ID: \"runbook-digest:page-18:chunk-2:v3\", Score: 0.84},\n        {ID: \"policy-digest:page-4:chunk-1:v3\", Score: 0.79},\n    }\n    answer := GeneratedAnswer{\n        Text:      \"Production changes require the documented approval step.\",\n        Citations: []string{\"runbook-digest:page-18:chunk-2:v3\"},\n    }\n\n    validationErr := validateCitations(retrieved, answer.Citations)\n    disposition := \"released\"\n    if validationErr != nil {\n        disposition = \"withheld\"\n    }\n\n    event := AuditEvent{\n        RequestID:        \"req-7f31\",\n        OccurredAt:       time.Now().UTC(),\n        CorpusSnapshot:   \"corpus-a17\",\n        RetrievalPolicy:  \"hybrid-v3\",\n        QueryDigest:      digest(\"What approval is required for production changes?\"),\n        Retrieved:        retrieved,\n        Citations:        answer.Citations,\n        AnswerDigest:     digest(answer.Text),\n        CitationValid:    validationErr == nil,\n        FinalDisposition: disposition,\n    }\n\n    if err := json.NewEncoder(os.Stdout).Encode(event); err != nil {\n        fmt.Fprintln(os.Stderr, err)\n        os.Exit(1)\n    }\n    if validationErr != nil {\n        fmt.Fprintln(os.Stderr, validationErr)\n        os.Exit(2)\n    }\n}\n```\n\nRun it with:\n\n```\ngo run main.go\n```\n\nWithholding is intentional. If the user interface must always return something, return a non-answer such as “No supported answer was found in the indexed documents” and record that disposition; don't let an uncited fallback bypass the same release rule. The catch is that strict withholding is not suitable when the workflow explicitly values partial discovery over a supported answer, such as exploratory search where users inspect source pages themselves. In that case, label the result as search output, expose the passages, and don't represent generated prose as grounded.\n\nA retrieval migration is acceptable only if old and new paths can be compared through the same evidence contract. Feed both paths the same pinned corpus snapshot and approved question set, then compare candidate membership, citation validity, and disposition. Score values may not be comparable across implementations, so don't declare a migration safe because two arbitrary decimals are close; compare the decisions the scores produce and retain each policy version.\n\nThe same replay gives the 02:17 responder a map. Group the affected audit events by corpus snapshot and retrieval-policy version, then compare those groups with the prior healthy window. If an impossible citation ID is absent from its request's ordered retrieval set, investigate request correlation and the release validator. If the expected page is absent from every candidate set, move earlier in the pipeline: PDF extraction, chunk identity, snapshot publication, or retrieval policy. If the page appears in the candidates but the answer does not cite it, move later: prompt assembly and support evaluation. These paths have different owners and rollback decisions. A single “RAG quality” alarm is weak operationally because it asks someone to reconstruct all of them under time pressure.\n\nBuild the replay suite from approved question-to-source-page mappings. Measure candidate-set success separately from answer support: did retrieval include an acceptable page, and did each released claim remain inside its cited evidence? A fluent answer can hide poor retrieval, while a good candidate set can be squandered later. Keeping the stages separate makes the alert actionable and keeps component owners from arguing over one blended score.\n\nMigration exposes lock-in in a measurable way. If corpus snapshots, chunk identities, and audit events use an application-owned schema, the serving implementation can change without erasing historical meaning. If those identities exist only inside one index, migration requires preserving two interpretation systems or abandoning replay for older answers. Decide which loss is acceptable before ingestion starts.\n\nOne more trap deserves a short rule.\n\nDon't replay against “latest.” Replay against the snapshot that served the request.\n\nCapacity planning follows the evidence path. Estimate ingestion demand from changed PDF bytes and pages per revision, index size from chunks per page and representation size, query demand from peak requests and candidates considered, and ledger retention from events per request times retained fields. Load-test publication separately from query serving. A large documentation update and a query spike consume different resources, and one shared concurrency limit couples two failure domains that the architecture can keep apart.\n\nThe build-versus-buy decision then follows the evidence contract rather than preceding it. A managed component can remove index operations from the team's on-call load; a self-hosted component can give the team tighter control over placement, retention, and change timing. Neither choice supplies auditability by itself.\n\n| Boundary | Build and operate | Managed component | Decision test | \n|---|---|---|---|\n| PDF extraction and identity | Control digest and chunk rules; own parser drift | Reduce parser upkeep; accept the offered identity model | Can an unchanged source revision reproduce the same chunk IDs? | \n| Retrieval index | Control placement and rollout; own capacity and recovery | Transfer more operations; verify event export and version semantics | Can a served result be tied to a pinned snapshot? | \n| Evidence ledger | Control schema, retention, and access; own durability | Reduce storage operations; validate portability and deletion controls | Can an investigator replay the join after the serving system changes? | \n| Release validator | Keep the invariant in application code | Delegate only if behavior is testable and observable | Can unsupported output be withheld before it reaches a user? | \n\nStick with managed infrastructure when it exports the identifiers and version context the ledger requires, and the reduction in on-call work matters more than tuning freedom. Build or self-host the boundary when retention, placement, deterministic identity, or replay access cannot be met by the managed contract. A team with no capacity to operate an index should not self-host merely to gain theoretical control; an ownership boundary that routinely misses its SLO is not safer.\n\nOnly now is the alert definition defensible. Emit bounded state transitions such as `retrieval_started`, `retrieval_completed`, `citation_validation_completed`, `answer_released`, and `answer_withheld`. Outcome and policy version can be metric labels when their value sets are controlled. Request IDs, filenames, and chunk IDs should remain in traces or audit storage rather than metric labels because their value sets grow with requests and the corpus.\n\nThe instrumentation change is small: add the evidence-join result and its version context at the release boundary, then carry the same request ID through retrieval, validation, and disposition. The page can now say which invariant failed and which policy state served it. Earlier indicators — rising zero-result rate, a shift in candidate scores after a retrieval-policy change, or a snapshot falling behind its source manifest — belong in a non-paging warning path until replay evidence shows that they predict the release SLO.\n\nI'm not sure any universal numeric threshold for citation-valid answers is defensible. Traffic volume, the base rate of questions the PDFs can answer, and the cost of withholding all change the paging trade-off. What resolves that uncertainty is replaying known questions against real corpus snapshots, observing the failure distribution, and choosing a window that corresponds to an action the on-call can actually take.\n\nPage on sustained user harm: a breach in the objective for releasable, citation-valid answers. A single rejected answer should be withheld and counted, but paging on each rejection imposes a false-positive cost during sparse traffic and trains responders to ignore the signal.\n\nThis returns us to the page. A threshold set too tightly wakes the on-call for isolated, correctly withheld requests; one set too loosely lets a sustained grounding regression reach users before anyone acts. The right alert is the narrowest sustained breach that maps to a documented rollback, snapshot repair, or policy rollback — and the audit ledger is what tells the responder which action fits.", "url": "https://wpnews.pro/news/how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded", "canonical_source": "https://dev.to/rasmusberg6592/how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded-1pc", "published_at": "2026-10-21 00:36:38+00:00", "updated_at": "2026-10-21 00:49:24.729308+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Go"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded", "markdown": "https://wpnews.pro/news/how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded.md", "text": "https://wpnews.pro/news/how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded.txt", "jsonld": "https://wpnews.pro/news/how-to-build-retrieval-audit-logs-for-internal-engineering-pdfs-in-go-grounded.jsonld"}}