{"slug": "bm25er-search-local-files-like-you-re-searching-the-web-bm25", "title": "Bm25er - Search local files like you're searching the web (BM25 implemimplementation)", "summary": "A developer's review of the Zig-based local file search tool Bm25er highlights usability and robustness issues, including a missing usage message, a crash when reading binary files due to an unguarded 512-byte buffer in the tokenizer, and a file size limit that fails on large files like raylib headers. The reviewer suggests using buffered reads and ASCII checks to handle binary files safely.", "body_md": "Hey! I did try it out and took a look at the code. Overall, I think your code is very reasonable. It could be improved a few ways, but mostly I want to talk about some of the bigger issues I had.\n\n``` js\n    const search_keyword = if (args.len > 1) args[1] else std.process.exit(1);\n```\n\nWhen I cloned the repo, the first thing I tried was `$ zig build run`\n\nand the error message I got back didn’t help me understand what went wrong. I changed it to this for my convenience:\n\n``` js\n    const search_keyword = if (args.len > 1) args[1] else {\n        std.debug.print(\"usage: bm25er PATTERN [DIRECTORY]\\n\", .{});\n        std.process.exit(1);\n    };\n```\n\nSecond issue:\n\n``` bash\n$ zig build run -- bm25\nFile zig-out/bin/bm25er is too big abbasthread 256656 panic: reached unreachable code\n```\n\nFirst problem is that “File zig-out/bin/bm25er is too big abbas” needs a `\\n`\n\nat the end. Second:\n\n```\n        //TODO: replace 512 with something smaller when the tokenizer is fixed and also supports html. You can check the len and if it exceeds the array size allocate on arena too.\n        var tok_buffer: [512]u8 = undefined;\n        const tok_lower = std.ascii.lowerString(&tok_buffer, tok);\n```\n\nThe error came from `lowerString()`\n\nbecause there’s no protection against reading binary files and in some of them `tok`\n\nis larger than 512. Since you own the memory under `tok`\n\n, you could actually avoid the buffer here and modify the source directly. Or, if you do want to keep them separate, please add a check here.\n\n``` js\nconst f = std.Io.Dir.readFileAlloc(dir, io, file_name, gpa, .limited(1_000_000))\n```\n\nWhen I tried to scan one of my repos, a header file imported by raylib was too large. `readFileAlloc()`\n\nis a nice convenient function, and I agree with your decision to limit the amount of memory this consumes, but since you don’t know the size of the files in advance and need to read many files, a better way is to set aside a sufficiently large buffer (ex: 4096) and do buffered reads without allocation.\n\nexample using the reader interface: [Zig Cookbook](https://cookbook.ziglang.cc/01-01-read-file-line-by-line/)\n\nOn binary files: I’ve heard of a technique for detecting if you’re reading a binary file where you check the first 128 or so bytes to see if they’re all valid, printable characters. As far as I know there isn’t support in the standard for non-ASCII but just checking `std.ascii.isAscii()`\n\nwould be okay.", "url": "https://wpnews.pro/news/bm25er-search-local-files-like-you-re-searching-the-web-bm25", "canonical_source": "https://ziggit.dev/t/bm25er-search-local-files-like-youre-searching-the-web-bm25-implemimplementation/17158#post_2", "published_at": "2026-08-09 22:28:34+00:00", "updated_at": "2026-08-09 22:36:23.787291+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Bm25er", "Zig", "raylib"], "alternates": {"html": "https://wpnews.pro/news/bm25er-search-local-files-like-you-re-searching-the-web-bm25", "markdown": "https://wpnews.pro/news/bm25er-search-local-files-like-you-re-searching-the-web-bm25.md", "text": "https://wpnews.pro/news/bm25er-search-local-files-like-you-re-searching-the-web-bm25.txt", "jsonld": "https://wpnews.pro/news/bm25er-search-local-files-like-you-re-searching-the-web-bm25.jsonld"}}