{"slug": "intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai", "title": "Intro to C++ and Rust: Just Enough to Read the Fast Code Under AI", "summary": "A tutorial post demonstrates how to read C++ and Rust code underlying AI libraries such as NumPy (written in C), PyTorch (C++ plus CUDA), Hugging Face tokenizers (Rust), llama.cpp (C and C++), Polars (Rust), Pydantic version 2 (Rust core), and Ruff and uv (Rust), arguing that Python orchestrates the work while compiled languages do the heavy lifting. The author compiled the C++ examples with g++ 13 on Ubuntu 24.04 but did not compile the Rust snippets, which the post labels standard idioms to verify before trusting. The post frames native code as a last resort after better algorithms, existing libraries, and NumPy vectorization, and notes that step can still disappoint.", "body_md": "Python is the language of AI, but it isn't the language doing the heavy lifting. Underneath NumPy, PyTorch, and many tools you'll use sits C, C++, Rust, and CUDA. You don't need to master them, but you should be able to read them.\n\nThis post is optional, and it's about reading, not writing. By the end you'll recognize the main features of C++ and Rust code, understand why native code is fast (and when it isn't), and have called a C++ function from Python yourself.\n\nA note on what I ran. The C++ code was compiled with g++ 13 on Ubuntu 24.04, and the outputs and compiler messages are real. I had no Rust toolchain in my test environment, so the Rust snippets are standard idioms that I did *not* compile. Treat them as reading material, and compile them yourself before trusting them.\n\n## Why the fast parts aren't written in Python\n\nBack in Post 15, we saw that Python's flexibility costs speed, and that libraries get around it by doing the heavy work in compiled code while Python just gives the orders. Some well-known examples:\n\n| Project | What it is | Core written in | \n|---|---|---|\n| NumPy | Arrays and math | C | \n| PyTorch | Deep learning | C++ (plus CUDA for GPUs) | \n| Hugging Face `tokenizers` | Fast tokenization | Rust | \n| llama.cpp | Running language models locally | C and C++ | \n| Polars | Data frames | Rust | \n| Pydantic (version 2) | Data validation | Rust core | \n| Ruff and uv | A linter, and a package manager | Rust | \n\nSo when you hit a confusing bug or a performance mystery inside one of these, the answer often lives in code that isn't Python. Reading it won't be your daily job, but being *able* to will set you apart.\n\nYou'll rarely need to *write* it. The ladder from Post 15 still applies: a better algorithm first, then a library that already does the job, then NumPy vectorization, and only then native code. As you'll see below, even that last step can disappoint.\n\n## What's different about C++ and Rust\n\nThree differences explain almost everything that feels strange:\n\n1. **They're compiled.** You don't run the source directly. A compiler translates it into machine code first, and it refuses to build if it finds certain mistakes. That's why a type error stops you*before* the program runs.\n2. **Types are fixed, and written out.** Every variable has a declared type (`int` ,`float` ,`std::string` ), and it can't change.\n3. **You think about memory.** In Python, memory is looked after for you (Post 15). In C++ and Rust you decide who owns data, and who may look at it or change it.\n\n|  | Python | C++ | Rust | \n|---|---|---|---|\n| Typing | Dynamic | Static | Static | \n| Memory | Automatic | You manage it (modern C++ helps a lot) | The compiler enforces rules | \n| Mistakes like out-of-range access | Raise an error | Can silently misbehave | Caught, or panic safely | \n| Build tool | none / `pip` ,`uv` | `g++` ,`make` , CMake | `cargo` | \n| Learning curve | Gentle | Wide and old, with a lot of history | Steep at first, strict compiler | \n\n## Reading C++\n\nHere's the cycle. You write `file.cpp`, compile it, and run the result:\n\n```\ng++ -std=c++20 -Wall -Wextra -O2 basics.cpp -o basics\n./basics\n```\n\n`-Wall -Wextra` turns on helpful warnings, and `-O2` turns on optimization, which matters a great deal (more on that below).\n\nHere's a program that packs in most of the syntax you'll meet when reading AI code. Skim it first, and then I'll explain the cues:\n\n```\n#include <algorithm>\n#include <iostream>\n#include <string>\n#include <unordered_map>\n#include <vector>\n\nstruct Message {\n    std::string role;\n    std::string content;\n    int tokens = 0;\n};\n\nint estimate_tokens(const std::string& text) {\n    return static_cast<int>(text.size() + 3) / 4;    // roughly 4 characters per token\n}\n\nvoid add_one(int& x) {    // a reference: changes the caller's variable\n    x += 1;\n}\n\ntemplate <typename T>\nT sum(const std::vector<T>& values) {\n    T total = 0;\n    for (const T& v : values) {\n        total += v;\n    }\n    return total;\n}\n\nint main() {\n    int count = 3;\n    double price = 0.5;\n    std::string name = \"Asha\";\n    std::cout << \"Hello, \" << name << \"! count=\" << count << \" price=\" << price << \"\\n\";\n\n    std::vector<int> numbers = {1, 2, 3, 4};\n    numbers.push_back(5);\n    std::cout << \"size=\" << numbers.size() << \" first=\" << numbers[0] << \" sum=\" << sum(numbers) << \"\\n\";\n    std::cout << \"float sum=\" << sum(std::vector<float>{0.5f, 1.5f}) << \"\\n\";\n\n    add_one(count);\n    int* p = &count;      // a pointer: the address of count\n    *p = *p + 10;         // follow the pointer\n    std::cout << \"count after add_one and pointer: \" << count << \"\\n\";\n\n    std::vector<std::string> words = {\"the\", \"cat\", \"the\", \"dog\"};\n    std::unordered_map<std::string, int> counts;\n    for (const auto& word : words) {\n        counts[word] += 1;\n    }\n    std::cout << \"the=\" << counts[\"the\"] << \" cat=\" << counts[\"cat\"] << \"\\n\";\n\n    Message m{\"user\", \"How do generators work?\"};\n    m.tokens = estimate_tokens(m.content);\n    std::cout << m.role << \": \" << m.content << \" (\" << m.tokens << \" tokens)\\n\";\n\n    struct Doc { std::string id; float score; };\n    std::vector<Doc> docs = {{\"a\", 0.31f}, {\"b\", 0.92f}, {\"c\", 0.55f}};\n    std::sort(docs.begin(), docs.end(), [](const Doc& x, const Doc& y) { return x.score > y.score; });\n    for (const auto& d : docs) {\n        std::cout << d.id << \":\" << d.score << \" \";\n    }\n    std::cout << \"\\n\";\n\n    try {\n        std::vector<int> small = {1, 2, 3};\n        std::cout << small.at(10) << \"\\n\";\n    } catch (const std::out_of_range& e) {\n        std::cout << \"caught: \" << e.what() << \"\\n\";\n    }\n    return 0;\n}\n```\n\nIt prints:\n\n```\nHello, Asha! count=3 price=0.5\nsize=5 first=1 sum=15\nfloat sum=2\ncount after add_one and pointer: 14\nthe=2 cat=1\nuser: How do generators work? (6 tokens)\nb:0.92 c:0.55 a:0.31 \ncaught: vector::_M_range_check: __n (which is 10) >= this->size() (which is 3)\n```\n\n### The reading cues\n\nYou don't need to write any of that from memory. You need to recognize what you're looking at:\n\n| You see | It means | \n|---|---|\n| `#include <vector>` | Like `import` : pull in a library header | \n| `std::` | \"From the standard library,\" like a module prefix. `std::string` ,`std::vector` | \n| `int count = 3;` | The **type comes first** , then the name, and every statement ends with`;` | \n| `{ ... }` | Blocks use braces, not indentation | \n| `std::vector<int>` | A list of ints. The `<...>` is a**template** parameter, like Python's`list[int]` | \n| `template <typename T>` | A function written once that works for any type `T` . Generics (Post 25) | \n| `const` | \"This won't be changed.\" Everywhere, and a good sign | \n| `int&` /`const std::string&` | A **reference** : another name for the same variable, with no copy | \n| `int* p = &count;` | A **pointer** .`&count` is \"the address of count,\" and`*p` is \"the thing it points to\" | \n| `p->field` | The same as `(*p).field` : reach through a pointer | \n| `auto` | \"Let the compiler work out the type\" | \n| `[](...) { ... }` | A **lambda** , an inline function like Python's`lambda` , but with a full body | \n| `for (const auto& x : items)` | Python's `for x in items` , without copying each item | \n| `std::cout << ...` | Print. Think of it as `print` | \n\nThe reference rule is the one worth understanding properly, because it's all about the copying trap from Post 5. By default, C++ passes arguments **by value**, meaning it copies them. Passing a million-float vector by value copies a million floats. So you'll see `const std::vector<float>&` constantly: \"a reference to the vector, which I promise not to change,\" with no copy and no cost. And a non-`const` reference like `int& x` is how a function changes its caller's variable, as `add_one` did above.\n\n### Memory and the danger of silence\n\nYou rarely see raw memory management in modern C++. A `std::vector` owns its memory and frees it automatically when it goes out of scope. But C++ still lets you do dangerous things quietly. If you write `small[10]` on a 3-item vector, there's no error. It reads whatever happens to be in memory at that spot: garbage, or a crash, or something that looks right and isn't. That's called **undefined behavior**, and it's the source of countless bugs and security problems. The checked version, `.at(10)`, raises an exception, which is what the last lines of the program above demonstrate.\n\nI'm not going to run the dangerous version for you, because the whole point is that its behavior can't be predicted. Remember it when you read C++: where Python would raise an `IndexError`, C++ may not tell you anything went wrong.\n\n### What compiler errors look like\n\nThe compiler is strict, and its messages can be long, but the first error usually has the answer. Two real ones:\n\n``` js\nbroken.cpp:3:17: error: invalid conversion from 'const char*' to 'int' [-fpermissive]\n    3 |     int count = \"three\";\n```\n\nYou tried to put text into an `int`. And a missing semicolon:\n\n```\nbroken2.cpp:3:22: error: expected ';' before 'return'\n    3 |     std::cout << \"hi\"\n```\n\nRead the first error, fix it, and recompile. Later errors are often just fallout from the first one.\n\n## Reading Rust\n\nRust has the same speed goals as C++, with a very different way of staying safe. I'll keep this short, and remember that I haven't compiled these snippets.\n\nHere's the dot product and cosine similarity from Post 15 in Rust:\n\n``` php\nfn dot(a: &[f32], b: &[f32]) -> f32 {\n    a.iter().zip(b).map(|(x, y)| x * y).sum()\n}\n\nfn norm(v: &[f32]) -> f32 {\n    dot(v, v).sqrt()\n}\n\nfn cosine(a: &[f32], b: &[f32]) -> f32 {\n    dot(a, b) / (norm(a) * norm(b))\n}\n\nfn main() {\n    let a = vec![1.0, 2.0, 3.0];\n    let b = vec![4.0, 5.0, 6.0];\n    println!(\"dot = {}\", dot(&a, &b));\n    println!(\"cosine = {:.4}\", cosine(&a, &b));\n}\n```\n\nThat should print `dot = 32` and `cosine = 0.9746` (I checked those numbers separately in Python). Here's how to read it:\n\n| You see | It means | \n|---|---|\n| `fn dot(a: &[f32], b: &[f32]) -> f32` | A function. Names come first, then `: type` , and`-> f32` is the return type, like Python hints | \n| `&[f32]` | A **borrowed slice** of floats: a read-only view, with no copy | \n| `a.iter().zip(b).map(\\|(x, y)\\| x * y).sum()` | An **iterator chain** . Close to Python's`sum(x * y for x, y in zip(a, b))` | \n| `\\|x\\| x * 2` | A closure, Rust's lambda | \n| no `return` , no`;` on the last line | The last expression is the return value | \n| `let a = ...` /`let mut a = ...` | Variables are **immutable** unless you say`mut` | \n| `vec![1.0, 2.0]` | Build a `Vec` , Rust's growable list | \n| `println!(...)` | A macro (the `!` gives it away) | \n| `use std::collections::HashMap;` | Import. Rust's dictionary is `HashMap` | \n\nA few more features come up all the time:\n\n```\nstruct Message {\n    role: String,\n    content: String,\n    tokens: Option<u32>,          // a value that might be missing\n}\n\nimpl Message {\n    fn token_estimate(&self) -> u32 {\n        self.tokens.unwrap_or((self.content.len() as u32 + 3) / 4)\n    }\n}\n\nfn parse_port(text: &str) -> Result<u16, std::num::ParseIntError> {\n    let port: u16 = text.trim().parse()?;     // ? means: on error, return it right away\n    Ok(port)\n}\n```\n\n- `struct` plus`impl` is Rust's version of a class (Post 9): data in one block, methods in another.\n- `Option<T>` replaces`None` . A value is either`Some(x)` or`None` , and the compiler*forces* you to handle both. There's no equivalent of the`NoneType has no attribute` error from Post 11.\n- `Result<T, E>` replaces exceptions. A function that can fail returns either`Ok(value)` or`Err(error)` , and`?` passes errors upward.\n- `match` is like a more powerful`if/elif` that must cover every case, much like the exhaustive`switch` from Post 25.\n\n### Ownership: the idea behind Rust\n\nRust's big idea is **ownership**. Every value has exactly one owner, and when the owner goes away, the value is freed. You can lend a value out with `&` (read-only, many at once) or `&mut` (writable, only one at a time). The compiler checks all of this *before* the program runs, which is how Rust can be fast with no garbage collector and still avoid many memory bugs. Here's the famous error:\n\n``` js\nlet a = vec![1, 2, 3];\nlet b = a;                  // ownership MOVES to b\nprintln!(\"{:?}\", a);        // error: borrow of moved value: `a`\n```\n\nAfter `let b = a`, `a` is no longer valid. If you see an error like that, it isn't a bug in your logic. Rust is telling you that you used something after giving it away. The usual fixes are to borrow (`&a`) or to copy deliberately (` a.clone()`). It's the strictest part of the language, and the reason Rust feels steep at first.\n\nRust projects use **Cargo**: `cargo new myproject` creates one, `cargo run` builds and runs, and `cargo build --release` produces the optimized version. (Like C++'s `-O2`, **release mode matters**: Rust's default debug builds can be many times slower.)\n\n## The performance reality check\n\nNow the part that surprised me, and that I think matters most. Native code is fast, but \"I wrote it in C++\" does not mean \"it's faster than what I had.\" I ran four experiments, all in C++ on my small test machine. Treat the numbers as examples, since yours will differ.\n\n### 1. Compiler flags matter a lot\n\nThe same dot product of two 50-million-float arrays, compiled three ways:\n\n```\n-O0                 149 ms\n-O2                  51 ms\n-O3 -march=native    57 ms\n```\n\nWithout optimization (`-O0`, the default if you forget), it's about three times slower than with `-O2`. Always benchmark with optimization on.\n\n### 2. The comparison with NumPy\n\nFor the same dot product, a plain Python loop would take about **5 seconds** (I timed a million items at 0.1 seconds and scaled up). My C++ took about 51 ms, roughly a hundred times faster. And NumPy's `np.dot` took **24 ms**, which is *twice as fast as my C++*. That tells you something from Post 19: this job is **memory-bound**. There's very little math per number, so the limit is how fast data can be read from memory, and NumPy's version is tuned to do exactly that well.\n\n### 3. Fast and wrong\n\nWhen I first printed the C++ result, something looked off. Each dot product should be 50,000,000 × 3 = 150,000,000. The simple loop with a `float` accumulator gave:\n\n```\nexpected:            150000000\nfloat accumulator:   67108864\ndouble accumulator:  150000000\nNumPy float32 np.dot: 150000000.0\n```\n\nThe fast loop was **silently wrong**, and not by a little. The cause is Post 19's lesson on precision. A `float` has about seven digits of precision, so once the running total gets large, adding a small number to it changes nothing, and the sum stalls. Check: `float32` can't distinguish 16,777,217 from 16,777,216. Here's the fix, a `double` accumulator, in a function that otherwise reads the same:\n\n``` js\nfloat dot_float(const float* a, const float* b, int n) {\n    float sum = 0.0f;\n    for (int i = 0; i < n; ++i) sum += a[i] * b[i];\n    return sum;                      // wrong for long inputs\n}\n\ndouble dot_double(const float* a, const float* b, int n) {\n    double sum = 0.0;\n    for (int i = 0; i < n; ++i) sum += static_cast<double>(a[i]) * b[i];\n    return sum;                      // correct\n}\n```\n\nNumPy got it right because it adds numbers in a smarter order. The lesson is the one from Post 15: **always check that the fast version gives the right answer.** Nothing crashed, nothing warned, and the timing looked great.\n\n### 4. Memory order matters, again\n\nHere's the memory-locality idea from Post 19, in C++. Multiplying two matrices takes three nested loops, and you can nest them in different orders. Both versions do exactly the same arithmetic:\n\n```\n// i-j-k order: B is read by jumping n elements at a time\nvoid matmul_ijk(const float* A, const float* B, float* C, int n) {\n    for (int i = 0; i < n; ++i)\n        for (int j = 0; j < n; ++j) {\n            float sum = 0.0f;\n            for (int k = 0; k < n; ++k) sum += A[i * n + k] * B[k * n + j];\n            C[i * n + j] = sum;\n        }\n}\n\n// i-k-j order: B is read in order\nvoid matmul_ikj(const float* A, const float* B, float* C, int n) {\n    for (int i = 0; i < n; ++i)\n        for (int k = 0; k < n; ++k) {\n            float a = A[i * n + k];\n            for (int j = 0; j < n; ++j) C[i * n + j] += a * B[k * n + j];\n        }\n}\n```\n\nFor 768 × 768 matrices, the first took about **627 ms** and the second about **222 ms**, with exactly the same answer (the biggest difference was 0). Same work, three times faster, only because the second one walks through memory in order. And NumPy's `A @ B` took **6 ms**, about 100 times faster than the first version and 35 times faster than the second. Libraries like it use techniques such as splitting the matrices into blocks that fit in cache, and special instructions that process several numbers at once. This is why \"just rewrite it in C++\" so often disappoints, and why the heavy math in AI always goes through tuned libraries (and GPUs).\n\nThe takeaway from all four: native code *can* be much faster than Python, but speed comes from understanding memory, precision, and the hardware, not from the language. When a library already does the job, use it.\n\n## Calling native code from Python\n\nSo when *is* it worth writing? When no library does exactly what you need, and the profile (Post 15) says that's where your time goes. Python can call compiled code in several ways, and I'll show the simplest, which needs nothing installed.\n\nThe basic recipe: write the function with a plain **C interface** (`extern \"C\"`), build it into a **shared library** (a `.so` file on Linux, `.dylib` on macOS, `.dll` on Windows), and load it with Python's built-in `ctypes` module. Other tools do the same job with more comfort: `cffi`; `pybind11` and `nanobind` for C++; Cython; and for Rust, PyO3 with a build tool called maturin. (I haven't used those here, so check their documentation.)\n\nTwo properties of `ctypes` are worth knowing. It does **no checking whatsoever**: it trusts that you passed the right types, as we'll see go wrong in a moment. And it releases Python's GIL (Post 14) while the C function runs, so several threads can run native code at the same time.\n\n## Reading unfamiliar native code\n\nWhen you open a C++ or Rust file in an AI project, here's a field guide:\n\n- **Start from the edges.** Find the function that Python calls (the bindings) or`main` , and work inward. Read the**signatures** (names, parameters, return types) before any function bodies. They tell the story.\n- **Understand the data layout.** A tensor in native code is usually a pointer to numbers plus a**shape** , a**dtype** , and**strides** (how far to jump to reach the next row). That's the memory picture from Post 19.\n- **Find the hot loop.** Somewhere there's the loop that runs millions of times. Everything else is setup.\n- **Ignore the noise.** Heavy template syntax, macros, and long type names look scary. Skim them, and read the logic.\n- **Recognize GPU code.** A function marked`__global__` and launched with`<<<blocks, threads>>>` is a CUDA kernel. We'll look at those in Post 70.\n- **Use the tools you already have.** Build with debug symbols, add prints, read the tests, and use`git log -p` and`git blame` (Post 22) to see*why* a line exists.\n\nOn safety: C++ can have data races when threads share data, and out-of-range accesses that fail silently. Safe Rust rules out data races and out-of-range memory errors at compile time. An `unsafe` block in Rust means \"the compiler can't check this, so I promise it's fine.\" In both languages, that's where to look first when something strange happens.\n\n## Should you learn to write it?\n\n| If you... | Then | \n|---|---|\n| Build apps and pipelines on top of models | Reading ability is plenty. Stay in Python and TypeScript | \n| Hit a hot spot that profiling proves | Try a library, then NumPy, then Numba or Cython, before writing C++ or Rust | \n| Work on inference engines, custom GPU kernels, or tokenizers | Learn C++ (and CUDA), or Rust, properly. It's the job | \n| Want to write safe, fast tools and libraries | Rust is a great choice, and the ecosystem is growing | \n| Work on a project that's already C++ | Learn C++ | \n\nIf you do pick one: C++ has an enormous existing codebase in ML and GPU work, so it's the one you'll most often *need* to read. Rust has a gentler path to writing correct code once you get past the compiler's strictness, and it's a popular choice for newer tools. Both are long learning journeys. Neither is required to become an excellent AI engineer.\n\n## Mini project: a C++ search function called from Python\n\nLet's do the full loop. We'll take the cosine search from Post 15, write the scoring in C++, build it as a shared library, call it from Python, check it against NumPy, and measure. Create `fastsearch.cpp`:\n\n```\n#include <cmath>\n#include <vector>\n\nextern \"C\" {\n\n// scores[q * n_docs + d] = cosine similarity between query q and document d.\n// All arrays are float32, stored row by row.\nvoid cosine_scores(const float* docs, int n_docs,\n                   const float* queries, int n_queries,\n                   int dim, float* scores) {\n    std::vector<double> doc_norms(n_docs);\n    for (int d = 0; d < n_docs; ++d) {\n        double sum = 0.0;\n        for (int i = 0; i < dim; ++i) {\n            double x = docs[d * dim + i];\n            sum += x * x;\n        }\n        doc_norms[d] = std::sqrt(sum);\n    }\n\n    for (int q = 0; q < n_queries; ++q) {\n        const float* query = queries + q * dim;\n        double query_sum = 0.0;\n        for (int i = 0; i < dim; ++i) {\n            query_sum += static_cast<double>(query[i]) * query[i];\n        }\n        double query_norm = std::sqrt(query_sum);\n\n        for (int d = 0; d < n_docs; ++d) {\n            const float* doc = docs + d * dim;\n            double dot = 0.0;\n            for (int i = 0; i < dim; ++i) {\n                dot += static_cast<double>(doc[i]) * query[i];\n            }\n            scores[q * n_docs + d] = static_cast<float>(dot / (doc_norms[d] * query_norm));\n        }\n    }\n}\n\n}  // extern \"C\"\n```\n\nNotice what we learned going in: `double` accumulators, so the sums stay accurate, and all arrays passed as plain pointers with their sizes as separate arguments. That's the \"pointer plus shape\" layout from the field guide. The `extern \"C\"` block gives the function a plain, unmangled name that Python can find.\n\nBuild it into a shared library (`-shared -fPIC` makes a loadable library):\n\n```\ng++ -std=c++20 -Wall -Wextra -O2 -shared -fPIC fastsearch.cpp -o libfastsearch.so\n```\n\nNow the Python side. Create `use_fastsearch.py`:\n\n``` python\nimport ctypes\nimport time\n\nimport numpy as np\n\nlib = ctypes.CDLL(\"./libfastsearch.so\")\nlib.cosine_scores.argtypes = [\n    ctypes.c_void_p, ctypes.c_int,    # docs, n_docs\n    ctypes.c_void_p, ctypes.c_int,    # queries, n_queries\n    ctypes.c_int,                     # dim\n    ctypes.c_void_p,                  # scores (output)\n]\nlib.cosine_scores.restype = None\n\ndef cosine_scores_cpp(docs, queries):\n    docs = np.ascontiguousarray(docs, dtype=np.float32)\n    queries = np.ascontiguousarray(queries, dtype=np.float32)\n    scores = np.empty((len(queries), len(docs)), dtype=np.float32)\n    lib.cosine_scores(\n        docs.ctypes.data, len(docs),\n        queries.ctypes.data, len(queries),\n        docs.shape[1],\n        scores.ctypes.data,\n    )\n    return scores\n\ndef cosine_scores_numpy(docs, queries):\n    docs = docs / np.linalg.norm(docs, axis=1, keepdims=True)\n    queries = queries / np.linalg.norm(queries, axis=1, keepdims=True)\n    return queries @ docs.T\n\ndef best_time(func, *args, repeats=10):\n    times = []\n    for _ in range(repeats):\n        start = time.perf_counter()\n        func(*args)\n        times.append(time.perf_counter() - start)\n    return min(times)\n\nrng = np.random.default_rng(0)\nfor n_docs in (2_000, 100_000):\n    docs = rng.standard_normal((n_docs, 128)).astype(np.float32)\n    queries = rng.standard_normal((20, 128)).astype(np.float32)\n\n    from_cpp = cosine_scores_cpp(docs, queries)\n    from_numpy = cosine_scores_numpy(docs, queries)\n    same_top3 = (np.argsort(-from_cpp, axis=1)[:, :3] == np.argsort(-from_numpy, axis=1)[:, :3]).all()\n\n    t_cpp = best_time(cosine_scores_cpp, docs, queries, repeats=5)\n    t_np = best_time(cosine_scores_numpy, docs, queries, repeats=5)\n    print(f\"{n_docs:>7,} docs: C++ {t_cpp * 1000:8.2f} ms | NumPy {t_np * 1000:8.2f} ms | \"\n          f\"max difference {np.abs(from_cpp - from_numpy).max():.1e} | same top 3: {same_top3}\")\n\n# the silent-mistake demo: hand ctypes the wrong dtype\ndocs64 = rng.standard_normal((2_000, 128))          # float64 by default!\nqueries64 = rng.standard_normal((20, 128))\nwrong = np.empty((20, 2_000), dtype=np.float32)\nlib.cosine_scores(docs64.ctypes.data, 2_000, queries64.ctypes.data, 20, 128, wrong.ctypes.data)\nright = cosine_scores_numpy(docs64.astype(np.float32), queries64.astype(np.float32))\nprint(\"float64 passed straight in: max difference from the right answer =\", f\"{np.abs(wrong - right).max():.2f}\")\n```\n\nRun `python use_fastsearch.py`. On my machine it printed:\n\n```\n2,000 docs: C++     3.81 ms | NumPy     0.52 ms | max difference 1.8e-07 | same top 3: True\n100,000 docs: C++   197.86 ms | NumPy    36.53 ms | max difference 2.4e-07 | same top 3: True\nfloat64 passed straight in: max difference from the right answer = nan\n```\n\nThree things to take from it.\n\n**It's correct.** The C++ scores match NumPy's to about one part in ten million, which is just `float32` rounding, and the top three results are identical for every query.\n\n**It's much faster than pure Python, and slower than NumPy.** In Post 15, pure Python took about 0.65 seconds for the 2,000-document case. The C++ version took 3.8 milliseconds, around 170 times faster. But NumPy took 0.5 milliseconds, about seven times faster than my C++. NumPy turns the whole job into one matrix multiplication handled by a heavily tuned library, while my version is straightforward loops. Compiler flags help (with `-O3 -march=native -ffast-math` the 100,000-document case dropped to about 75 ms, and with `-O0` it was about 765 ms), but `-ffast-math` lets the compiler rearrange floating-point math, so results can change slightly, and it still didn't catch NumPy.\n\n`ctypes` **checked nothing.** In the last test, I passed `float64` arrays straight in, without converting them to `float32`. There was no error and no crash. The function just read 4-byte floats out of memory that actually held 8-byte doubles, and produced garbage, printed as `nan`. That's why the wrapper function calls `np.ascontiguousarray(..., dtype=np.float32)` first: it guarantees the exact type and memory layout the C++ code expects. With raw pointers, the types are a promise *you* keep.\n\n### Make it yours\n\n1. Write a test (Post 13) that compares `cosine_scores_cpp` with NumPy on random inputs, and that deliberately fails if you remove the dtype conversion.\n2. Do the same work with the loop order changed, so the inner loop walks through memory in order. Does it help here?\n3. Add top-k selection inside C++ with `std::partial_sort` , so Python only receives the top three document numbers.\n4. Try `-fopenmp` to spread the work across cores (`#pragma omp parallel for` on the outer loop). It needs OpenMP support, so check your compiler.\n5. Rewrite the `dot` functions in Rust, build them as a`cdylib` , and call them from Python with`ctypes` .\n6. Look up `cblas_sgemm` . Calling a real BLAS library from your C++ is how you'd get NumPy-like speed.\n\n## Practice\n\nAbout 20 minutes of real effort on each before asking for help.\n\n1. Compile and run the C++ example, then change one thing at a time: pass `sum` a vector of`double` s, break a type on purpose, and read the compiler's error.\n2. Read a small open-source C++ or Rust file (the source of a tokenizer or a small utility works well). Write down each function signature in plain English before reading any body.\n3. Benchmark a function at `-O0` ,`-O2` , and`-O3` . Explain the differences you see.\n4. Reproduce the `float` versus`double` accumulator bug with your own numbers. At about what total does the`float` sum stall?\n5. Change the matrix multiplication to a third loop order (j-k-i) and measure it. Predict first, and explain the result using the memory picture from Post 19.\n6. Write a `C++` function that takes a`const std::vector<int>&` and returns the largest value, and a version that takes the vector by value. What would you measure to show the difference?\n7. Translate the Rust `Message` and`parse_port` snippets into Python, using`dataclass` and exceptions. What does the Rust version force you to handle?\n8. Use `git log -p` and`git blame` on a file in an open-source native library, and find the commit that explains a surprising line.\n\nThen ask an AI tutor to review your answers, and don't let it rewrite them.\n\n## What to take away\n\n- The fast parts of AI are mostly C, C++, Rust, and CUDA, and Python gives the orders. You'll read this code far more often than you write it.\n- C++ and Rust are compiled, statically typed, and make you think about memory. Learn the reading cues: `std::` ,`<T>` ,`&` ,`*` ,`const` ,`auto` , lambdas, and in Rust`fn` ,`let mut` ,`Option` ,`Result` ,`match` , and`?` .\n- Pass big things by reference. In C++, out-of-range access can fail silently, and Rust's ownership rules catch many of those mistakes before the program runs.\n- Always benchmark with optimization on, and always check that the fast version gives the right answer. A `float` accumulator silently produced the wrong total.\n- Your first C++ is probably not faster than NumPy. Speed comes from memory order, precision, and tuned libraries, not from the language.\n- `ctypes` plus`extern \"C\"` is the simplest bridge from Python, and it trusts you completely about types and memory layout.\n- Read native code from its edges: the bindings, then the signatures, the data layout, and the hot loop.\n\n**Previous:** Post 25, *TypeScript for AI Engineers*\n\n**Next in the series:** Post 27, *Mini Project 1*, where we build the first Phase 1 project: a command-line tool, such as a markdown note searcher.", "url": "https://wpnews.pro/news/intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai", "canonical_source": "https://zyvop.com/intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai-4af5t", "published_at": "2026-11-11 06:11:54+00:00", "updated_at": "2026-11-11 06:19:52.182535+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "artificial-intelligence"], "entities": ["Python", "C++", "Rust", "NumPy", "PyTorch", "Hugging Face", "llama.cpp", "Polars"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai", "markdown": "https://wpnews.pro/news/intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai.md", "text": "https://wpnews.pro/news/intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai.txt", "jsonld": "https://wpnews.pro/news/intro-to-c-and-rust-just-enough-to-read-the-fast-code-under-ai.jsonld"}}