{"slug": "semantic-search-in-dart-without-the-hand-written-cosine-loop", "title": "Semantic search in Dart without the hand-written cosine loop", "summary": "A Flutter developer built vector_kit, a Dart library that accelerates semantic search by storing embeddings as a packed Float32List and using SIMD dot products, achieving 11x faster top-k cosine similarity queries over 20,000 documents compared to a naive List<List<double>> loop. The library also offers an int8 quantized matrix that reduces memory from 29.3 MB to 7.6 MB while maintaining 100% recall@10 on the demo data.", "body_md": "A packed matrix and SIMD dot products for top-k similarity, with the memory numbers spelled out.\n\nI had a Flutter app with about 20,000 short documents and a 384-dimension embedding for each one. Take the user's query embedding, find the five closest documents by cosine similarity, show them. On device, no server round trip.\n\nThe first version is the obvious one. Embeddings in a `List<List<double>>`\n\n, a loop that scores every row, sort, take the top five.\n\n``` js\ndouble cosine(List<double> a, List<double> b) {\n  var dot = 0.0, na = 0.0, nb = 0.0;\n  for (var i = 0; i < a.length; i++) {\n    dot += a[i] * b[i];\n    na += a[i] * a[i];\n    nb += b[i] * b[i];\n  }\n  return dot / (sqrt(na) * sqrt(nb));\n}\n\nList<int> topK(List<double> query, List<List<double>> rows, int k) {\n  final scored = <MapEntry<int, double>>[];\n  for (var i = 0; i < rows.length; i++) {\n    scored.add(MapEntry(i, cosine(query, rows[i])));\n  }\n  scored.sort((x, y) => y.value.compareTo(x.value));\n  return scored.take(k).map((e) => e.key).toList();\n}\n```\n\nThis is correct. It is also 15.9 ms per query over the 20,000-row index on my machine (Apple Silicon, Dart 3.11). At 15.9 ms you feel it when the user is typing and you want to re-rank on each keystroke, and it grows linearly with the corpus.\n\nThe cost is not the algorithm. A linear scan is the right algorithm at this size. The cost is the layout. `List<List<double>>`\n\nis a list of pointers to separate heap objects, each `double`\n\nis a boxed 64-bit float, and the inner loop chases pointers and reads memory that was never meant to be walked in order.\n\n`vector_kit`\n\nstores the whole index as one contiguous `Float32List`\n\nand scores it with SIMD. Same linear scan, same exact results, different memory.\n\n```\nimport 'package:vector_kit/vector_kit.dart';\n\n// rows: List<List<double>> of 20000 x 384, from your embedding model.\nfinal index = VectorMatrix.fromRows(rows);\n\nfinal query = model.embed(userText); // List<double>, length 384\nfinal hits = index.topKCosine(query, 5);\n\nfor (final hit in hits) {\n  print('${hit.index}  score=${hit.score}');\n}\n```\n\n`topKCosine`\n\nover the 20,000-row index, top-5, runs in 1.4 ms. That is the same 15.9 ms work from above, about 11x faster, and the ranking is identical because nothing is approximated.\n\nOne detail that saved me a wrapper: `query`\n\nis a plain `List<double>`\n\n. The embedding that comes out of a model is already a `List<double>`\n\n, and it goes straight into `topKCosine`\n\nwith no conversion to `Float32List`\n\nat the call site. The matrix is packed once when you build it, not on every query.\n\nThe primitive underneath is a `dot`\n\nthat maps to hardware SIMD. At 768 dimensions it is 142 ns per call, against 665 ns for the equivalent loop over two `List<double>`\n\n. The raw product is exposed if that is all you need:\n\n```\nfinal a = Float32List.fromList(embA);\nfinal b = Float32List.fromList(embB);\nfinal d = dot(a, b); // 142 ns at 768 dims\n```\n\nThe gap holds at larger scale. `topKCosine`\n\nwith k=10 over 100,000 rows is 13.3 ms. The full-scan-and-sort baseline over the same data is 82 ms. The packed version never materializes 100,000 score entries into a list and sorts them. It keeps a running top-k, so it does less allocation as well as less arithmetic.\n\nThe 20,000 x 384 index in float32 is 29.3 MB. That is the number you have to budget for, because it sits in memory for the life of the feature, and on a phone 29.3 MB is not free.\n\n`vector_kit`\n\nhas an int8 `QuantizedMatrix`\n\nthat holds the same index in 7.6 MB, a quarter of the float32 size.\n\n```\nfinal quant = QuantizedMatrix.from(index);\nfinal hits = quant.topKCosine(query, 10);\n```\n\nQuantization is lossy, so the question is what it costs in ranking quality. I measured recall against the float ranking instead of assuming it. On the demo's data the int8 index returned 100% recall@10, meaning the top-10 set matched the float top-10 exactly. That number is data-dependent. If your embeddings are less separated you will lose some recall, so run the same measurement on your own corpus before you ship. The tool to do that is in the box. The guarantee is not.\n\nThis is a linear scan. Every row is scored on every query. That is why the results are exact and why there is no index to build or tune beyond packing the matrix. It also sets a ceiling.\n\nUp to somewhere around 100k to 1M vectors, scanning is fine and the numbers above are what you get. Past that, into tens of millions of vectors, a linear scan is the wrong structure no matter how tight the inner loop is, and you want an approximate nearest neighbor index (HNSW, IVF) that trades exactness for sublinear query time. `vector_kit`\n\ndoes not do that and does not pretend to. At that scale, reach for a real ANN library or a vector database and pay for the recall tuning that comes with it.\n\nIt is also not a full linear algebra package. It does dot products, cosine similarity, and top-k over a packed matrix. For general matrix multiply, decompositions, or autograd, this is not that.\n\nFor the case it targets, an exact top-k over a corpus that fits in memory on the device, the two numbers that matter are 1.4 ms per query and the choice between 29.3 MB and 7.6 MB for the index. Those are the ones I would check against your own data first.", "url": "https://wpnews.pro/news/semantic-search-in-dart-without-the-hand-written-cosine-loop", "canonical_source": "https://dev.to/yusufihsangorgel/semantic-search-in-dart-without-the-hand-written-cosine-loop-26e7", "published_at": "2026-07-21 00:20:09+00:00", "updated_at": "2026-07-21 00:29:29.060204+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["vector_kit", "Flutter", "Dart", "Apple Silicon"], "alternates": {"html": "https://wpnews.pro/news/semantic-search-in-dart-without-the-hand-written-cosine-loop", "markdown": "https://wpnews.pro/news/semantic-search-in-dart-without-the-hand-written-cosine-loop.md", "text": "https://wpnews.pro/news/semantic-search-in-dart-without-the-hand-written-cosine-loop.txt", "jsonld": "https://wpnews.pro/news/semantic-search-in-dart-without-the-hand-written-cosine-loop.jsonld"}}