Bm25er - Search local files like you're searching the web (BM25 implemimplementation)

Hey! I did try it out and took a look at the code. Overall, I think your code is very reasonable. It could be improved a few ways, but mostly I want to talk about some of the bigger issues I had.

    const search_keyword = if (args.len > 1) args[1] else std.process.exit(1);

When I cloned the repo, the first thing I tried was $ zig build run

and the error message I got back didn’t help me understand what went wrong. I changed it to this for my convenience:

    const search_keyword = if (args.len > 1) args[1] else {
        std.debug.print("usage: bm25er PATTERN [DIRECTORY]\n", .{});
        std.process.exit(1);
    };

Second issue:

$ zig build run -- bm25
File zig-out/bin/bm25er is too big abbasthread 256656 panic: reached unreachable code

First problem is that “File zig-out/bin/bm25er is too big abbas” needs a \n

at the end. Second:

        //TODO: replace 512 with something smaller when the tokenizer is fixed and also supports html. You can check the len and if it exceeds the array size allocate on arena too.
        var tok_buffer: [512]u8 = undefined;
        const tok_lower = std.ascii.lowerString(&tok_buffer, tok);

The error came from lowerString()

because there’s no protection against reading binary files and in some of them tok

is larger than 512. Since you own the memory under tok

, you could actually avoid the buffer here and modify the source directly. Or, if you do want to keep them separate, please add a check here.

const f = std.Io.Dir.readFileAlloc(dir, io, file_name, gpa, .limited(1_000_000))

When I tried to scan one of my repos, a header file imported by raylib was too large. readFileAlloc()

is a nice convenient function, and I agree with your decision to limit the amount of memory this consumes, but since you don’t know the size of the files in advance and need to read many files, a better way is to set aside a sufficiently large buffer (ex: 4096) and do buffered reads without allocation.

example using the reader interface: Zig Cookbook

On binary files: I’ve heard of a technique for detecting if you’re reading a binary file where you check the first 128 or so bytes to see if they’re all valid, printable characters. As far as I know there isn’t support in the standard for non-ASCII but just checking std.ascii.isAscii()

would be okay.