Bm25er - Search local files like you're searching the web (BM25 implemimplementation)

A developer's review of the Zig-based local file search tool Bm25er highlights usability and robustness issues, including a missing usage message, a crash when reading binary files due to an unguarded 512-byte buffer in the tokenizer, and a file size limit that fails on large files like raylib headers. The reviewer suggests using buffered reads and ASCII checks to handle binary files safely.

Hey I did try it out and took a look at the code. Overall, I think your code is very reasonable. It could be improved a few ways, but mostly I want to talk about some of the bigger issues I had. js const search keyword = if args.len 1 args 1 else std.process.exit 1 ; When I cloned the repo, the first thing I tried was $ zig build run and the error message I got back didn’t help me understand what went wrong. I changed it to this for my convenience: js const search keyword = if args.len 1 args 1 else { std.debug.print "usage: bm25er PATTERN DIRECTORY \n", .{} ; std.process.exit 1 ; }; Second issue: bash $ zig build run -- bm25 File zig-out/bin/bm25er is too big abbasthread 256656 panic: reached unreachable code First problem is that “File zig-out/bin/bm25er is too big abbas” needs a \n at the end. Second: //TODO: replace 512 with something smaller when the tokenizer is fixed and also supports html. You can check the len and if it exceeds the array size allocate on arena too. var tok buffer: 512 u8 = undefined; const tok lower = std.ascii.lowerString &tok buffer, tok ; The error came from lowerString because there’s no protection against reading binary files and in some of them tok is larger than 512. Since you own the memory under tok , you could actually avoid the buffer here and modify the source directly. Or, if you do want to keep them separate, please add a check here. js const f = std.Io.Dir.readFileAlloc dir, io, file name, gpa, .limited 1 000 000 When I tried to scan one of my repos, a header file imported by raylib was too large. readFileAlloc is a nice convenient function, and I agree with your decision to limit the amount of memory this consumes, but since you don’t know the size of the files in advance and need to read many files, a better way is to set aside a sufficiently large buffer ex: 4096 and do buffered reads without allocation. example using the reader interface: Zig Cookbook https://cookbook.ziglang.cc/01-01-read-file-line-by-line/ On binary files: I’ve heard of a technique for detecting if you’re reading a binary file where you check the first 128 or so bytes to see if they’re all valid, printable characters. As far as I know there isn’t support in the standard for non-ASCII but just checking std.ascii.isAscii would be okay.