{"slug": "turning-a-pile-of-documents-into-a-searchable-useable-knowledge-base", "title": "Turning a pile of documents into a searchable useable knowledge base", "summary": "DocuBrowse, a new desktop application, turns local document collections into searchable knowledge bases using AI-powered semantic search and summaries, all running offline with no internet or API keys required. The tool supports multiple file formats, offers hybrid keyword and semantic search, and includes PII protection features.", "body_md": "Packaged for Linux (RPM, DEB, tarball), Windows (zip), and macOS (dmg). Interfaces are stable; breaking changes are avoided where possible.\n\n**DocuBrowse turns a messy pile of documents into something you can actually search.**\nPoint it at your files — PDFs, ebooks, Word docs, notes, whatever — and it builds a smart\nindex that understands not just keywords, but meaning. Ask for \"that contract about the\nlease renewal\" and find it even if those exact words never appear. Click any result for\nan instant AI summary before you even open the file. PII aware and works with multiple document directories.\n\nDocuBrowse runs entirely on your own machine using local AI models — no internet\nconnection required, no accounts, no API keys, and no per-query costs eating into a\ntoken budget. **Your data. Your AI.**\n\nUnder the hood: SQLite FTS5 keyword search plus AI-powered semantic similarity and synopsis generation (Ollama + nomic-embed-text + dolphin3). Supports multiple document and source code types.\n\n**Keyword Search**— fast full-text search via SQLite FTS5 (title, author, subject, tags, snippet)** Semantic Search**— AI-powered similarity via Ollama embeddings (nomic-embed-text:latest)** Hybrid Mode**(default) — 70% semantic + 30% keyword, merged and re-ranked\n\n- Click any document title for a Kindle-style book-jacket synopsis, generated on demand\nvia Ollama (\n`dolphin3:latest`\n\n) and cached in the database after first generation. Semantic search embeddings are produced by a second local model (`nomic-embed-text:latest`\n\n)\n\n**Formats**: PDF, DOCX, PPTX, XLSX, EPUB, MOBI, AZW3, AZW, HTML, TXT, Markdown** PDF intelligence**: pdfplumber (preferred) with pypdf fallback for bloated-object files;`layout=False`\n\nretry for complex layouts; scanned (image-only) PDFs detected and routed to`ocr_list_pdfs.txt`\n\n**Word documents**: python-docx extracts paragraphs, tables, and core properties (title, author, subject)** Presentations**: python-pptx extracts slide text, notes, and core properties** Spreadsheets**: openpyxl extracts cell values and sheet names** E-books**: ebooklib for EPUB; mobi package + Calibre fallback for MOBI/AZW3; DRM-encrypted AZW files indexed with metadata only (title/author visible, body not searchable)**Metadata**: title, author, subject extracted from document metadata fields; auto-generated tags from directory structure and content keywords**PII protection**: post-ingest scanner detects SSN, credit card, bank routing/account number, DOB, MRN, driver license, passport patterns; removes matching documents and permanently blacklists them\n\n- Dark/Light theme toggle\n- Paginated results (50 docs/page) with Back/Next controls\n- Alphabetic index bar (A–Z, 0–9) for quick navigation, with state preserved across page loads; Home button to reset\n- Tag cloud for filtering by topic\n- Relevance score badges (0–100%) on every result\n**Open button** on each result card — launches the file in your default app via`xdg-open`\n\n- Click document title for an AI synopsis; 📋 copies path to clipboard; 🗑 deletes file from disk and index (with confirmation)\n- Moved/deleted documents: clicking a doc whose file no longer exists shows a dismissable modal (and removes it from the index on dismiss) if its filesystem is mounted, or a toast (no index change) if the filesystem can't be verified (e.g. unmounted drive)\n\n- General panel: document directory (with live directory browser), any number of additional scan directories (added/removed under the same panel — automatically included in\n`scan`\n\n/`rescan`\n\n, no extra command needed), working directory, and port - Ignored Directories panel: browse to add a directory to\n`ignore_dirs.txt`\n\n, with a confirmation prompt before purging already-indexed documents under it, and a confirmation before removing an entry\n\n- Search latency: <150ms typical\n- Parallel PDF extraction with\n`ProcessPoolExecutor`\n\n(physical-core-aware worker count) - Memory-safe: kernel-enforced RLIMIT_AS (6 GB/worker) + pause/resume on free-RAM threshold\n\nClick any thumbnail to view full size.\n\n| Dark Mode | Light Mode |\n|---|---|\n\n| Settings | AI Synopsis |\n|---|---|\n\nSettings is a standalone page at\n\n`/settings`\n\n(opened in a new tab via the gear icon). The General panel covers the document directory (with live directory browser, plus any number of additional scan directories), working directory, and port; the Ignored Directories panel manages scan exclusions, each with a directory browser, add/clear controls, and confirmation before removal.\n\n- Python 3.9+\n`pdfplumber`\n\n,`pypdf`\n\n— PDF extraction`python-docx`\n\n— Word documents`python-pptx`\n\n— PowerPoint presentations`openpyxl`\n\n— Excel spreadsheets`ebooklib`\n\n,`beautifulsoup4`\n\n,`mobi`\n\n— E-books`psutil`\n\n— cross-platform process and hardware detection**Calibre**— E-book metadata and conversion (required for MOBI/AZW3/AZW indexing):`sudo dnf install calibre`\n\nor`sudo apt install calibre`\n\n- Ollama — installed automatically by\n`docubrowser start`\n\nif missing - Modern browser (Chrome, Firefox, Safari, Edge)\n\nSee [INSTALL.md](/linuxrebel/DocuBrowser/blob/main/INSTALL.md) for a full step-by-step guide.\n\nDocuBrowse ships as RPM, DEB, tarball, Windows zip, and macOS dmg packages.\nDownload the appropriate package from the\n[Releases](https://github.com/linuxrebel/DocuBrowser/releases) page.\n\n```\n# Fedora / RHEL\nsudo dnf install ./docubrowser-foss-0.9.0-7.noarch.rpm\n\n# Debian / Ubuntu / Mint\nsudo apt install ./docubrowser-foss_0.9.0-7_all.deb\n\n# Any Linux (tarball)\ntar xzf docubrowser-foss-0.9.0-7.tar.gz\ncd docubrowser-foss-0.9.0-7\nsudo ./install.sh\n```\n\n**Windows:** Extract the zip, then double-click `Install.bat`\n\n. Requires\nPython 3.9+ and Ollama to be pre-installed. Installs to\n`%USERPROFILE%\\DocuBrowse`\n\nwith a Start Menu shortcut — no admin required.\nYou may need to log out and back in for the shortcut to appear.\n\n**macOS:** Open the dmg, then double-click `Install.command`\n\n(right-click →\nOpen the first time — the scripts are unsigned). Requires Python 3.9+.\nInstalls to `~/Applications/DocuBrowse/`\n\nwith a Python virtualenv, CLI\nwrappers at `/usr/local/bin/docubrowser`\n\nand `/usr/local/bin/docuback`\n\n(sudo prompted; falls back to `~/bin/`\n\nif declined), and a `DocuBrowse.app`\n\nlauncher that starts the server and opens the web UI in Terminal.\n\nAll Linux methods install to `/opt/docubrowser/`\n\nwith a Python virtualenv,\nCLI wrappers at `/usr/bin/docubrowser`\n\nand `/usr/bin/docuback`\n\n, a desktop\nmenu entry under Office, and all Python dependencies from `requirements.txt`\n\n.\n\nOnce installed, the CLI is the `docubrowser`\n\ncommand — drop the `./`\n\nand `.py`\n\nfrom every example below. For example, `docubrowser start`\n\nand\n`docubrowser rescan`\n\n. The `./docubrowser.py <cmd>`\n\nform shown throughout the\nrest of this README is the dev / cloned-repo path (running directly out of a\ncheckout).\n\nTo uninstall: `sudo dnf remove docubrowser-foss`\n\n(RPM),\n`sudo apt remove docubrowser-foss`\n\n(DEB), `sudo ./uninstall.sh`\n\n(tarball),\ndouble-click `Uninstall.bat`\n\n(Windows), or double-click `Uninstall.command`\n\n(macOS — on the dmg or in `~/Applications/DocuBrowse/`\n\n).\n\n```\ncd /path/to/DocuBrowse\n\n# Scan and index your documents\n./docubrowser.py rescan\n\n# Start the server\n./docubrowser.py start\n\n# Open the UI\n./docubrowser.py open\n```\n\nOn an installed system, use the\n\n`docubrowser`\n\ncommand instead, e.g.`docubrowser rescan`\n\n/`docubrowser start`\n\n/`docubrowser open`\n\n.\n\n`docubrowser.py start`\n\nautomatically verifies Ollama is installed, running, and has\nboth required models — `nomic-embed-text:latest`\n\n(embeddings) and\n`dolphin3:latest`\n\n(synopsis generation) — installing/starting/pulling\nas needed.\n\n```\nUsage: docubrowser.py <command> [options]\n```\n\n| Command | Description |\n|---|---|\n`start` |\nStart the search server (runs Ollama check first) |\n`stop` |\nStop the server |\n`restart` |\nStop then start |\n`status` |\nShow server status, document count, embedding count, tag count |\n`scan [TYPE ...]` |\nScan and index documents (no embedding) |\n`rescan [TYPE ...]` |\nScan + generate embeddings |\n`scan-file --file PATH` |\nExtract and index a single file, then embed it |\n`embed` |\nGenerate/refresh embeddings for un-embedded documents |\n`open` |\nOpen the DocuBrowse UI in your default browser |\n`purge` |\nScan index for PII and remove matching documents |\n`ignore add|remove|list DIR` |\nManage directories excluded from scanning (auto-purges on add) |\n`report` |\nWalk doc directory and show file-type breakdown (no DB changes) |\n`scan-missing [--db PATH] [--dry-run]` |\nOpt-in cleanup: classify every indexed path as present/missing/unmounted, delete `missing` rows (cascades), leave `unmounted` rows alone |\n`stopall` |\nStop all running scans, embeds, and the server |\n`duplist` |\nList duplicate documents (exact SHA256 + optional near-duplicate) |\n`dupclean` |\nInteractive TUI to review and remove duplicate documents |\n\n```\n--db PATH      SQLite database path (overrides config)\n--port PORT    Server port (overrides config)\n--config FILE  Config file path\n# Server management\n./docubrowser.py start\n./docubrowser.py start --port 9000\n./docubrowser.py status\n./docubrowser.py stop\n./docubrowser.py stopall\n\n# Scanning\n./docubrowser.py scan                          # scan all supported types\n./docubrowser.py scan pdf                      # PDFs only\n./docubrowser.py scan pdf txt                  # PDFs and plain text\n./docubrowser.py scan --limit 100              # first 100 unindexed files only\n./docubrowser.py rescan                        # scan + embed all types\n./docubrowser.py rescan pdf --workers 4        # PDFs only, 4 workers\n./docubrowser.py rescan --no-embed             # scan without embedding step\n./docubrowser.py rescan --doc-dir /data/docs\n\n# Single-file indexing (useful for retrying blacklisted files)\n./docubrowser.py scan-file --file /path/to/document.pdf\n./docubrowser.py scan-file --file /path/with spaces/doc.pdf   # no quoting needed\n./docubrowser.py scan-file --file /path/to/doc.pdf --no-embed\n\n# Reporting and maintenance\n./docubrowser.py report                         # file-type breakdown, no DB changes\n./docubrowser.py embed                          # embed any un-embedded docs\n./docubrowser.py purge --dry-run               # preview PII matches (safe)\n./docubrowser.py purge                         # remove PII documents (prompts)\n\n# Excluding directories from scanning\n./docubrowser.py ignore add /mnt/data/Documents/myWorkDocs   # exclude + purge indexed docs under it\n./docubrowser.py ignore list                                  # show ignored directories\n./docubrowser.py ignore remove /mnt/data/Documents/myWorkDocs # re-allow (rescan to re-index)\n\n# Duplicate detection and cleanup\n./docubrowser.py duplist                       # find exact SHA256 duplicates\n./docubrowser.py duplist --near-dups           # also find near-duplicates (cosine ≥97%)\n./docubrowser.py duplist --near-dups --threshold 0.95\n./docubrowser.py dupclean                      # interactive Keep A/Keep B/Keep Both TUI\n./docubrowser.py dupclean --near-dups          # include near-duplicates in cleanup\n\n# Cleaning up moved/deleted documents (opt-in, not run automatically)\n./docubrowser.py scan-missing --dry-run        # report counts only, no DB changes\n./docubrowser.py scan-missing                  # delete rows for genuinely-missing files\nTypes: pdf  txt  md  html  (default: all four)\n\nExamples:\n  rescan pdf             PDFs only\n  rescan pdf txt         PDFs and plain text\n  rescan                 all supported types (prompts if unfiltered)\n```\n\n`scan-file`\n\nis designed for retrying individual problem files:\n\n- Removes the file from\n`scan_blacklist.txt`\n\nif listed (explicit retry) - Refuses files in\n`pii_blacklist.txt`\n\n(permanent PII block) - Detects scanned (image-only) PDFs → adds to\n`ocr_list_pdfs.txt`\n\n- Paths with spaces work without quoting:\n`--file`\n\naccepts multiple tokens and rejoins them\n\nDocuBrowse reads the first config file it finds:\n\n`/etc/docubrowse.config`\n\n(system-wide)`./docubrowse.config`\n\n(next to`docubrowser.py`\n\n)\n\nIf neither exists, built-in defaults apply — except `doc_dir`\n\n, which has no\ndefault. Until a document directory is configured (via the Settings gear icon\nin the web UI, or by setting `doc_dir`\n\nin `docubrowse.config`\n\n), the web UI\nshows a banner prompting you to configure one, and CLI commands that need a\ndocument directory (`rescan`\n\n, `report`\n\n, `scan`\n\n) exit with an error explaining\nhow to set it.\n\n```\n# docubrowse.config\ndoc_dir      = /mnt/data/Documents\ndb_path      = /home/user/DocuBrowse/du-docs.db\nport         = 8643\nwork_dir     = /home/user/DocuBrowse\n```\n\n| Key | Default |\n|---|---|\n`doc_dir` |\n(none — must be configured via Settings or docubrowse.config) |\n`db_path` |\n`<script dir>/du-docs.db` |\n`port` |\n`8643` |\n`work_dir` |\n`<script dir>` |\n\n```\n┌─────────────────────────────────────┐\n│  docubrowser.py  (CLI entry point)  │\n│  ensure_ollama.py (prereq check)    │\n└──────────┬──────────────────────────┘\n           │ subprocess / direct call\n           ↓\n┌──────────────────────┐  ┌─────────────────────────────────┐\n│  scan_docs.py        │  │  doc_search.py  (HTTP :8643)    │\n│  ProcessPoolExecutor │  │  GET /  /api/search             │\n│  pdf_extractor.py    │  │  GET /api/stats  /api/tags      │\n│  embed_docs.py       │  │  GET /api/open  /api/config     │\n│                      │  GET /api/delete  /api/synopsis │\n└──────────┬───────────┘  └──────────────┬──────────────────┘\n           │                              │\n           └──────────┬───────────────────┘\n                      ↓\n        ┌──────────────────────────────────────────────────┐\n        │  du-docs.db  (SQLite FTS5)                       │\n        │  Ollama (nomic-embed-text + dolphin3)  │\n        └──────────────────────────────────────────────────┘\n```\n\n| Script | Role |\n|---|---|\n`docubrowser.py` |\nCLI launcher — all commands |\n`ensure_ollama.py` |\nChecks/installs Ollama binary, service, and required models |\n`doc_search.py` |\nHTTP server; search API and UI |\n`docubrowse_db.py` |\nSQLite schema and migrations |\n`platform_paths.py` |\nCross-platform path resolution and process management |\n`scan_docs.py` |\nDocument discovery, extraction, and DB writes |\n`pdf_extractor.py` |\nPDF-specific extraction with pdfplumber/pypdf |\n`docx_extractor.py` |\nWord document extraction (python-docx) |\n`ebook_extractor.py` |\nEPUB/MOBI/AZW3/AZW extraction (ebooklib + Calibre) |\n`hardware_utils.py` |\nCPU/GPU/RAM detection, worker count formula |\n`embed_docs.py` |\nSends text to Ollama; stores 768-dim vectors |\n`purge_pii.py` |\nScans index for PII; removes and blacklists matches |\n`dup_detect.py` |\nExact (SHA256) and near-duplicate (cosine similarity) detection |\n\n| File | Purpose | Permanent? |\n|---|---|---|\n`scan_blacklist.txt` |\nFiles that failed extraction | No — remove line to retry |\n`pii_blacklist.txt` |\nFiles removed for containing PII | Yes — never re-ingest |\n`ocr_list_pdfs.txt` |\nImage-only PDFs needing OCR | N/A — informational |\n`ignore_dirs.txt` |\nDirectories excluded from scanning (managed via `ignore` command) |\nNo — `ignore remove` + `rescan` |\n\nBase URL: `http://localhost:8643`\n\n| Method | Path | Description |\n|---|---|---|\n`GET` |\n`/` |\nServe `index.html` (injects a per-process CSRF token) |\n`GET` |\n`/settings` |\nServe `settings.html` |\n`GET` |\n`/api/stats` |\nTotal docs, embedded count, unique tag count |\n`GET` |\n`/api/tags` |\nTag list with counts (≥3 occurrences) |\n`GET` |\n`/api/search` |\nSearch with pagination |\n`GET` |\n`/api/letters` |\nFirst-letter index for the alphabetic bar |\n`GET` |\n`/api/synopsis` |\nGenerate/return an AI synopsis for a document |\n`GET` |\n`/api/config` |\nCurrent server configuration |\n`GET` |\n`/api/ignore-dirs` |\nList excluded directories |\n`GET` |\n`/api/scan-dirs` |\nList additional scan directories |\n🔒 `GET` |\n`/api/browse` |\nDirectory browser for Settings (token-gated) |\n🔒 `POST` |\n`/api/open` |\nOpen a file with xdg-open/gio (validates against DB) |\n🔒 `POST` |\n`/api/delete` |\nDelete a file from disk and remove from index (path must be indexed) |\n🔒 `POST` |\n`/api/config` |\nSave server configuration |\n🔒 `POST` |\n`/api/ignore-dirs` |\nAdd/remove an excluded directory |\n🔒 `POST` |\n`/api/scan-dirs` |\nAdd/remove an additional scan directory |\n\n🔒 = state-changing or filesystem-exposing; requires the per-process\n`X-CSRF-Token`\n\nheader and a loopback `Origin`\n\n/`Referer`\n\n. The token is injected\ninto the served HTML, so only the first-party UI can call these. See\n[Security](#security). All requests are also rejected unless the `Host`\n\nheader\nis `localhost`\n\n/`127.0.0.1`\n\n/`[::1]`\n\n(DNS-rebinding protection).\n\nIf the indexed path no longer exists on disk, `/api/open`\n\nreturns one of:\n\n```\n{\"ok\": false, \"error\": \"missing\", \"message\": \"...\"}\n{\"ok\": false, \"error\": \"unmounted\", \"message\": \"...\"}\n```\n\n`missing`\n\nmeans the file's filesystem is mounted and the file is genuinely gone — the UI\nshows a dismissable modal and deletes the document from the index (and disk-adjacent DB\nrows) when dismissed. `unmounted`\n\nmeans the path's filesystem can't currently be verified\n(likely an unmounted drive) — the UI shows a toast and makes no DB changes. See\n`scan-missing`\n\nfor the equivalent batch cleanup.\n\n```\nGET /api/search?q=QUERY&offset=0&mode=both\n```\n\n| Param | Values | Default |\n|---|---|---|\n`q` |\nsearch string | `\"\"` (returns all docs) |\n`mode` |\n`both` | `keyword` | `semantic` |\n`both` |\n`offset` |\ninteger | `0` |\n\n```\n{\n  \"documents\": [\n    {\n      \"id\": 1,\n      \"name\": \"doc.pdf\",\n      \"title\": \"Document Title\",\n      \"author\": \"Jane Smith\",\n      \"subject\": \"Cloud Security\",\n      \"description\": \"First 500 chars of content...\",\n      \"path\": \"/mnt/data/Documents/doc.pdf\",\n      \"tags\": [\"pdf\", \"security\", \"cloud\"],\n      \"modified_at\": \"2026-06-07T14:30:00\",\n      \"score\": 0.95,\n      \"fts_score\": 0.8,\n      \"sem_score\": 0.98\n    }\n  ],\n  \"query\": \"cloud security\",\n  \"count\": 50,\n  \"total\": 312,\n  \"offset\": 0,\n  \"has_more\": true,\n  \"mode\": \"both\"\n}\n# Read endpoints are plain GETs:\ncurl \"http://localhost:8643/api/stats\"\ncurl \"http://localhost:8643/api/search?q=kubernetes&mode=both\"\ncurl \"http://localhost:8643/api/search?q=&offset=50\"\n```\n\nMutating endpoints (`/api/delete`\n\n, `/api/open`\n\n, the `POST`\n\nconfig/dir routes)\nand `/api/browse`\n\nrequire the per-process CSRF token and a loopback origin, so\nthey aren't easily exercised with a bare `curl`\n\n— drive them from the UI, or\npass `-X POST -H \"X-CSRF-Token: <token>\" -H \"Origin: http://localhost:8643\"`\n\nwhere `<token>`\n\nis read from the `<meta name=\"csrf-token\">`\n\ntag in `/`\n\n.\n\nA non-empty query is scored two ways and merged; metadata is then fetched only for the requested page (the whole corpus is no longer loaded per request).\n\n```\nfinal_score = 0.3 × keyword_score + 0.7 × semantic_score   (mode=both)\n```\n\n- Backed by the SQLite\n**FTS5** index via`MATCH`\n\n+`bm25()`\n\n— not a Python substring scan. - Query tokens are quoted and prefix-matched (\n`\"tok\"*`\n\n), OR-combined, so arbitrary input (operators, quotes,`C++`\n\n,`&`\n\n) can't break the query. - Per-column BM25 weights echo the old field priorities (name 6, title 8, author 7, subject 5, description 3, content_snippet 3, tags 4); the result is normalized to 0–1.\n- Orphan\n`doc_fts`\n\nrowids (contentless FTS has no FK cascade) are pruned against the live document set so they can't inflate totals.\n\n- Cosine similarity between the query embedding and each document embedding.\n- Computed against an\n**in-process, L2-normalized embedding matrix**(one vectorized NumPy matrix-vector product), cached and invalidated when the embeddings table changes — instead of reloading every BLOB per request. - Range 0.0–1.0; minimum threshold (semantic-only mode):\n**0.30**. - Embeddings: 768-dimensional float32 vectors (nomic-embed-text:latest).\n\nDocuBrowse binds to localhost and is intended for single-user local use, but it is hardened so a malicious web page you happen to visit can't reach it:\n\n**Host-header allow-list**— every request is rejected unless`Host`\n\nis`localhost`\n\n/`127.0.0.1`\n\n/`[::1]`\n\n(optionally with the serving port). Defeats DNS-rebinding against the localhost-bound server.**CSRF tokens on mutations**—`/api/delete`\n\nand`/api/open`\n\nare POST-only; they plus the POST config/dir routes and the filesystem-exposing`/api/browse`\n\nrequire a per-process`X-CSRF-Token`\n\n(injected into the served HTML, so only the first-party UI has it) and a loopback`Origin`\n\n/`Referer`\n\n.**No stored-data injection**— document fields are escaped for HTML and card actions use`data-*`\n\nattributes + delegated listeners (no inline`onclick`\n\nbuilt from document data), closing a stored-XSS vector.**PII purge** validates structurally before deleting: SSNs against SSA allocation rules, credit cards by length + issuer prefix + Luhn, bank routing numbers by ABA checksum + Federal Reserve prefix — so it both catches more real PII and avoids deleting docs over incidental number groups.\n\nThe server is localhost-only — it binds the loopback subnet and rejects all non-loopback connections at the socket level. No authentication is needed because only the local user can reach the server.\n\n```\nDocuBrowse/\n├── docubrowser.py          # CLI entry point (all commands)\n├── ensure_ollama.py        # Ollama prerequisite checker/installer\n├── doc_search.py           # HTTP search server (port 8643)\n├── docubrowse_db.py        # SQLite schema and migrations\n├── scan_docs.py            # Scanner: discovery, extraction, DB writes\n├── pdf_extractor.py        # PDF extraction (pdfplumber + pypdf fallback)\n├── docx_extractor.py       # Word document extraction (python-docx)\n├── ebook_extractor.py      # EPUB/MOBI/AZW3/AZW extraction (ebooklib + Calibre)\n├── hardware_utils.py       # CPU/GPU/RAM detection, worker formula\n├── embed_docs.py           # Embedding generation pipeline\n├── purge_pii.py            # PII scanner and purge tool\n├── dup_detect.py           # Exact (SHA256) and near-duplicate detection\n├── platform_paths.py       # Cross-platform paths and process management\n├── index.html              # Frontend UI (single-file, dark/light theme)\n├── du-docs.db              # SQLite database (gitignored)\n├── du-docs.db.example      # Empty schema for new installs\n├── scan_blacklist.txt      # Failed-extraction skiplist (gitignored)\n├── pii_blacklist.txt       # PII-removed files — permanent (gitignored)\n├── ocr_list_pdfs.txt       # Image-only PDFs needing OCR (gitignored)\n├── ignore_dirs.txt         # Directories excluded from scanning (gitignored)\n├── scan_dirs.txt           # Additional scan directories (gitignored)\n├── docubrowse.config       # Local config (optional, gitignored)\n├── INSTALL.md              # Step-by-step install guide\n├── README.md               # This file\n├── LICENSE                 # GPL-3.0\n├── packaging/              # RPM spec, DEB control, build scripts, installers\n│   ├── build_packages.sh   # Builds RPM, DEB, and tarball (Linux)\n│   ├── build_windows_zip.sh # Builds Windows zip\n│   ├── docubrowser-foss.spec  # RPM spec\n│   ├── docubrowser.desktop # Desktop menu entry (Linux)\n│   ├── install.sh          # Tarball installer (Linux)\n│   ├── uninstall.sh        # Tarball uninstaller (Linux)\n│   ├── windows/            # Windows installer/uninstaller scripts\n│   │   ├── Install.bat     # Double-click to install\n│   │   ├── Uninstall.bat   # Double-click to uninstall\n│   │   ├── install.ps1     # PowerShell installer\n│   │   └── uninstall.ps1   # PowerShell uninstaller\n│   └── macos/              # macOS installer/uninstaller scripts\n│       ├── build_macos_dmg.sh   # Builds the macOS dmg\n│       ├── Install.command      # Double-click to install\n│       └── Uninstall.command    # Double-click to uninstall\n├── systemd/\n│   └── docubrowser.service # systemd unit file\n├── status_docs/            # Project planning and decision logs\n│   ├── project_status.md   # Current version, session history\n│   └── DECISIONS.md        # Deferred decisions and known issues\n└── test_pdfs_live/         # 100 sample PDFs for testing\n```\n\nLarge document collections may trigger:\n\n```\nOSError: [Errno 28] inotify watch limit reached\n```\n\nThis is a Linux kernel limit, not a disk space issue. It's safe to ignore these warnings — but if you'd rather not see them, raise the limit for the duration of the scan:\n\n- Edit\n`/etc/sysctl.conf`\n\nand find the line:\n\n```\nfs.inotify.max_user_instances=128\n```\n\n- Raise it to\n`256`\n\nor`512`\n\n:\n\n```\nfs.inotify.max_user_instances=256\n```\n\n- Apply the change without rebooting:\n\n```\nsudo sysctl -p\n```\n\nOnce ingestion is finished, you can set the value back to `128`\n\n(edit the\nfile again and re-run `sudo sysctl -p`\n\n) — or just leave it raised.\n\nSome PDFs cause pdfminer to hang. These are auto-detected and blacklisted. If a specific file is causing problems, check:\n\n```\n# How many PDF objects does it have? (>8000 is abnormal)\npdfinfo /path/to/file.pdf | grep -i objects\n\n# Retry the file after it has been blacklisted\n./docubrowser.py scan-file --file /path/to/file.pdf\n```\n\nFiles with >8,000 PDF objects (usually caused by repeated ExifTool metadata updates) are automatically routed through pypdf instead of pdfminer.\n\nPDFs with no extractable text are detected and added to `ocr_list_pdfs.txt`\n\n. They are\nindexed with a placeholder (`[scanned PDF — OCR required]`\n\n) so they appear in browse\nbut won't match keyword or semantic searches until OCR is run.\n\nCheck the log:\n\n```\ntail -f /var/log/docubrowser.log\n# or\ntail -f ~/.local/share/docubrowser/docubrowser.log\nollama serve                       # start manually\nollama list                        # verify both models are present\nollama pull nomic-embed-text:latest              # embeddings, if missing\nollama pull dolphin3:latest                      # synopsis generation, if missing\n```\n\n| Limitation | Status |\n|---|---|\n| DRM-encrypted AZW not fully searchable | Metadata indexed; DeDRM_tools required for body text |\n| Scanned PDFs not searchable | Listed in ocr_list_pdfs.txt; OCR deferred |\n| Multiple top-level doc directories | Fully supported — configure any number of additional scan directories in the General panel (`scan_dirs.txt` ); `scan` /`rescan` automatically scan all of them into the single shared database |\n| Moved/renamed files | Not detected as moves — old path is removed (interactively or via `scan-missing` ), new path is picked up on next rescan as a fresh entry; true duplicates are caught by `duplist` /`dupclean` |\n| No authentication | Local use only; hardened against cross-origin/CSRF/DNS-rebinding (see\n|\n\n`status_docs/DECISIONS.md`\n\n)DocuBrowse is now packaged for Linux and Windows with native installers.\n\n**RPM, DEB, tarball, and Windows zip packages**—`build_packages.sh`\n\nproduces Linux packages;`build_windows_zip.sh`\n\nproduces the Windows zip. Linux installs to`/opt/docubrowser/`\n\nwith a Python virtualenv, CLI wrappers at`/usr/bin/docubrowser`\n\nand`/usr/bin/docuback`\n\n, and a desktop menu entry under Office. Windows installs to`%USERPROFILE%\\DocuBrowse`\n\nwith a Start Menu shortcut (no admin required).**Windows installer**—`Install.bat`\n\n/`install.ps1`\n\ndetects Python, creates a virtualenv, installs dependencies, and creates a Start Menu shortcut.`Uninstall.bat`\n\nreverses everything.**Cross-platform path abstraction**— new`platform_paths.py`\n\ncentralizes all runtime path selection (PID files, log files, backup directory) and process management (kill, find-by-script, kill-port). Linux paths are unchanged; Windows paths use`%USERPROFILE%\\DocuBrowse\\`\n\n.**Windows compatibility**— guarded all Unix-only constructs (`resource`\n\n,`SIGALRM`\n\n,`os.killpg`\n\n,`/proc`\n\naccess) behind platform checks. Process management uses`psutil`\n\nwith`/proc`\n\nfallback on Linux.**Backup/restore**—`backup_restore.py`\n\nsupports Windows privilege checks (`IsUserAnAdmin`\n\n) and handles missing`pwd`\n\nmodule gracefully.**Desktop menu entry**—`.desktop`\n\nfile uses`xdg-terminal-exec`\n\nfor reliable terminal launching across all desktop environments; categorized under Office.**Systemd service file**— included for system-level deployments on Linux.** dist/ pruning**— build script keeps only the latest 2 releases per format.** macOS dmg installer**—`packaging/macos/build_macos_dmg.sh`\n\nproduces a dmg with double-clickable`Install.command`\n\n/`Uninstall.command`\n\nscripts. Installs to`~/Applications/DocuBrowse/`\n\n(no sudo for the app itself) with a Python virtualenv, CLI wrappers, and a`DocuBrowse.app`\n\nlauncher whose icon is generated from`icons/icon-512.png`\n\nvia sips/iconutil.\n\nRemoves unused code paths and experimental features that were never part of the FOSS release, leaving a cleaner and more focused codebase.\n\n**Leaner server**—`doc_search.py`\n\nreduced by ~300 lines; removed unused network configuration, protocol negotiation, and handler code that had accumulated during development.**Leaner CLI**—`docubrowser.py`\n\nreduced by ~240 lines; removed the`setup-tls`\n\ncommand and related helpers that are not applicable to a localhost application.**Stale files removed**— deleted`branding.json.example`\n\nand other development-only files that were never used in production.**Documentation updated**— README, INSTALL, and architecture notes cleaned up to accurately reflect the current feature set.\n\n**Hide documents from view**— each card now has a 🙈 Hide icon that tags the document as \"hidden\" and fades it out of the listing. Hidden documents remain in the database and can be restored at any time.**\"Show 🙈\" toggle button**— added next to the page count in all views (All Documents, letter filter, search results). Clicking it reveals all hidden cards alongside normal ones; the button label switches to \"Hide 🙈\" to toggle them back off.**Unhide (👀) icon**— when hidden cards are visible, they display a 👀 icon instead of 🙈. Clicking it removes the \"hidden\" tag server-side, swaps the icon back to 🙈, and removes the \"hidden\" tag chip from the card.**New API endpoint:**— removes a single tag from a document. Parameters:`POST /api/remove-tag`\n\n`path`\n\n(URL-encoded file path),`tag`\n\n(tag name). Returns the updated tag list. CSRF-protected.**Card action icons restyled**— all icons (📋 🔖 🙈 ❌) now use solid, colorful emoji at full opacity. No more faded/dim icons in dark mode.[↑ Top](#top)\n\n**Dark & light mode palettes redesigned**— new CSS variable theming with`data-theme`\n\nattribute switching. Dark mode uses deep navy/purple tones with cyan, orange, and violet accents. Light mode uses clean whites with darkened accent variants for readability. Tag colors cycle through five distinct hues via`nth-child`\n\nselectors. Score badges, mode buttons, and action buttons all use the new palette.**Trash icon now opens a 4-option modal** instead of immediately deleting: (1) Remove from index only (file stays on disk, re-scanned next run), (2) Remove & blacklist (file stays, future scans skip it), (3) Remove & delete file from disk (with double-confirmation), (4) Cancel. Server API updated:`POST /api/delete?path=...&mode=db_only|blacklist|delete_file`\n\n(defaults to`db_only`\n\nfor backward compatibility).**Search scoring fixed in \"both\" mode**— keyword matches were previously buried under thousands of low-similarity semantic results. Now applies a semantic floor (`SEM_FLOOR=0.30`\n\n) and uses`max(fts, sem)`\n\nscoring instead of a weighted average that capped keyword-only hits at 0.3.(same as`scan`\n\ncommand now embeds by default`rescan`\n\n) — new installations get working semantic search out of the box. Added`--no-embed`\n\nand`--embed-workers`\n\nflags for opt-out.**Security hardening**— CSRF protection added to`/api/synopsis`\n\n; exception leak to client suppressed.\n\n**Open action button** replaces the old clickable file path link on each result card — launches the file in your default app via`xdg-open`\n\n.**Button styling** updated — buttons use accent-colored border and text with a filled hover state, replacing the previous dim/greyed appearance.\n\nwith current schema — fresh installs no longer hit an HTTP 500 (\"no such column: d.subject\") on first page load. The old example was built against an older schema (pre-author/subject/synopsis columns and pre-full-FTS5 index), causing the lazy migration to race against the first search request. New example has the correct schema from the start.`du-docs.db.example`\n\nregenerated\n\n**Settings moved to a standalone page**(`/settings`\n\n, opened in a new tab via the gear icon) — replaces the old modal. Full-width layout, a header \"Done\" button that saves config and returns to the search tab, and a redesigned Ignored Directories panel (description text, \"Add a directory to exclude\" row, \"Currently excluded directories\" list with inline ✕ remove buttons, confirm-before-purge on add and rescan-reminder on remove).**Alpha index bar (0-9, A-Z) is now a true global filter**— clicking a letter queries`/api/search?letter=X`\n\nfor*all*matching documents (not just the loaded page), paginated by the existing page-size preference; Next/Back and page-size changes work while filtered, and clicking the active letter again returns to All Documents. A \"Home\" button (left of \"0-9\") returns to All Documents from anywhere, and the index bar now persists across every view (All Documents, letter-filtered, search, paginated).**Multiple document directories confirmed fully automatic**:`resolve_doc_dirs()`\n\nunifies the configured docPath with`scan_dirs.txt`\n\ninto one ordered list;`scan`\n\n/`rescan`\n\nloop over every directory into the single shared database and run embedding once at the end — no manual per-directory rescan needed.- Removed the \"N embedded\" count from the header stats bar (now \"N docs · N tags\").\n`friendlyError()`\n\nhelper in`index.html`\n\ngives a clear \"Cannot reach the DocuBrowse service\" message (instead of a generic network error) if the server is down while a page is loaded — applied to search, filter, pagination, synopsis, open, and delete.- Synopsis modal's \"Generating synopsis...\" message now updates at 6s/25s with reassuring text so a slow cold-start Ollama request (up to ~90s) doesn't look hung.\n\n`doc_dir`\n\n/`docPath`\n\nno longer default to`/mnt/data/Documents`\n\n— an unconfigured document directory is now a valid state across the CLI (`docubrowser.py`\n\n), API (`doc_search.py`\n\n`/api/config`\n\n), and`install.sh`\n\n's generated config.- CLI commands that need a document directory (\n`rescan`\n\n,`report`\n\n,`scan`\n\n) now exit with a clear error pointing to the Settings gear,`docubrowse.config`\n\n, or`--doc-dir`\n\nif none is configured. `index.html`\n\nshows a banner — \"No document directory configured yet. Click the Settings (gear) icon...\" — whenever`/api/config`\n\nreports an empty`docPath`\n\n.- Added\n`uninstall.sh`\n\n, mirroring`install.sh`\n\n's user/system mode detection: stops/disables/removes the systemd unit, removes the CLI wrapper and install directory, cleans up pid/log files, and (system mode, separate confirmation) can remove the dedicated`docubrowse`\n\nuser/group.\n\n**Installer:** rewritten`install.sh`\n\n/`uninstall.sh`\n\nwith a clean user vs. system split — user mode installs to`~/.docubrowse`\n\n(own venv, wrapper at`~/.local/bin/docubrowser`\n\n, no root, no systemd); system mode installs to`/opt/docubrowse`\n\nas a dedicated`docubrowse`\n\nuser with a (not auto-enabled)`docubrowser.service`\n\nsystemd unit and`/usr/local/bin/docubrowser`\n\nwrapper.**Pre-flight checks:** the installer validates all prerequisites up front (python3 ≥ 3.9 + venv/ensurepip, rsync, curl, tar, calibre, ollama, plus getent/useradd/groupadd/systemctl in system mode) and reports everything missing at once before making any changes.**CLI:** the launcher is now installed as the`docubrowser`\n\ncommand (no`.py`\n\n).**requirements.txt** added and installed via`pip install -r requirements.txt`\n\n— now includes previously-missing deps (numpy, python-pptx, openpyxl) alongside pdfplumber, pypdf, python-docx, ebooklib, beautifulsoup4, mobi.**Fresh installs start empty:**`du-docs.db.example`\n\nnow ships empty, so new installs begin with no documents indexed.**Multiple document directories:** Settings now shows a single \"Document directories\" list (the old separate docPath + \"additional directories\" panels are merged).`rescan`\n\n/`scan`\n\nindex**every** listed directory; an explicit`--doc-dir`\n\nstill targets just one.`doc_dir`\n\nis now optional.\n\nRemediation of a full code-quality + security audit (details in\n`status_docs/DECISIONS.md`\n\n). Highlights:\n\n**Security:** Host-header allow-list (anti DNS-rebinding);`/api/delete`\n\nand`/api/open`\n\nmoved to POST and, with the POST config/dir routes and`/api/browse`\n\n, gated by a per-process CSRF token + loopback origin; stored-XSS vector closed (data-attribute + delegated listeners); PII purge now validates with SSA rules + Luhn/IIN.**Search:** keyword path now uses the FTS5`bm25()`\n\nindex and semantic scoring uses a cached NumPy embedding matrix instead of loading the whole corpus per request (keyword ~4ms, both ~55ms); server-side semantic search fixed (was silently returning nothing).**Reliability:**`INSERT … ON CONFLICT`\n\nupsert (re-indexing no longer wipes tags/embeddings/synopsis); worker-death \"suspect isolation\" blacklists only the real offender; schema init runs once per process; scan/embed commit on a ~2s time budget so the server isn't blocked;`dupclean`\n\nno longer corrupts the disk/DB on its main path; precise`/proc`\n\n-based worker termination; single shared document-delete helper; assorted medium/low fixes.**UI:** pagination Back/Next correct at all page sizes; a newer search now supersedes an in-flight page load (no stale results).\n\n`/api/open`\n\nnow returns`{\"ok\": false, \"error\": \"missing\"|\"unmounted\", \"message\": ...}`\n\nfor files that no longer exist, instead of a generic error.- The UI shows a dismissable modal for\n`missing`\n\nfiles (and removes them from the index on dismiss), or a toast for`unmounted`\n\nfiles (filesystem can't be verified, no index change). - New opt-in\n`scan-missing [--dry-run]`\n\nCLI command batch-cleans`missing`\n\nrows across the whole index without touching`unmounted`\n\nrows.\n\n- Fixed \"Open file\" (\n`/api/open`\n\n) silently doing nothing — the server's environment was missing`DBUS_SESSION_BUS_ADDRESS`\n\n/`DISPLAY`\n\n/`XAUTHORITY`\n\n/`XDG_RUNTIME_DIR`\n\n, so`xdg-open`\n\nexited successfully without launching the default app.`handle_open`\n\nnow reconstructs the desktop session environment and prefers`gio open`\n\nfor reliable launches.\n\n- ✅ DOCX extractor (python-docx)\n- ✅ EPUB/MOBI/AZW3/AZW extraction (ebooklib + Calibre)\n- No-extension file classification (magic bytes)\n- Scale to 10K+ documents\n\n- ✅\n`duplist`\n\n/`dupclean`\n\n— exact + near-duplicate detection and interactive cleanup - ✅ Config read/write via Settings UI (port, docPath, workDir)\n- Sliding window ETA for progress bar\n- File-type filter in search UI\n\n- Config persistence\n- Advanced filtering (date range, type, author)\n- Result export (CSV/JSON)\n- OCR integration for scanned PDFs\n\n- API key authentication\n- Document similarity clustering\n- Docker deployment\n\nDocuBrowse is developed with Claude as an active coding partner. To resume a session with full context, load these files at the start:\n\n| File | Contents |\n|---|---|\n`.claude/CLAUDE.md` |\nProject rules, key files, hard-won lessons |\n`status_docs/project_status.md` |\nVersion, session history, what's in progress |\n`status_docs/DECISIONS.md` |\nDeferred decisions, known issues, rationale |\n\n```\n# Print all three for copy/paste into any AI assistant\ncat .claude/CLAUDE.md status_docs/project_status.md status_docs/DECISIONS.md\n```\n\nGNU General Public License v3.0 or later (GPL-3.0-or-later).\n\nCopyright (C) 2026 James Sparenberg\n\nSee [LICENSE](/linuxrebel/DocuBrowser/blob/main/LICENSE) or [https://www.gnu.org/licenses/gpl-3.0.html](https://www.gnu.org/licenses/gpl-3.0.html).\n\n**DocuBrowse v0.9.0** — Fast, local, AI-powered document search.", "url": "https://wpnews.pro/news/turning-a-pile-of-documents-into-a-searchable-useable-knowledge-base", "canonical_source": "https://github.com/linuxrebel/DocuBrowser", "published_at": "2026-07-08 20:37:29+00:00", "updated_at": "2026-07-08 20:54:16.331834+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "natural-language-processing", "ai-products", "ai-infrastructure"], "entities": ["DocuBrowse", "Ollama", "SQLite", "nomic-embed-text", "dolphin3", "pdfplumber", "python-docx", "Calibre"], "alternates": {"html": "https://wpnews.pro/news/turning-a-pile-of-documents-into-a-searchable-useable-knowledge-base", "markdown": "https://wpnews.pro/news/turning-a-pile-of-documents-into-a-searchable-useable-knowledge-base.md", "text": "https://wpnews.pro/news/turning-a-pile-of-documents-into-a-searchable-useable-knowledge-base.txt", "jsonld": "https://wpnews.pro/news/turning-a-pile-of-documents-into-a-searchable-useable-knowledge-base.jsonld"}}