Cost of running agent memory/vector search

Polign, a new vector database that separates index storage from compute, stores all 12.5 million English Wikipedia passages (81.7 GiB) in an S3 bucket for $2.28 per month, and a $55 spot instance handles a dozen searches per second, while a $97 spot instance handles hundreds per second at 4 cents per million searches, 25x less than the cheapest managed vendor. The benchmark shows Polign is cheapest at medium and high traffic, with Hit@10 accuracy of 0.47 consistent across compute profiles.

Blog blog.html · Benchmark Cost of running agent memory at scale I built Polign to separate the index vector or BM25 from the compute that serves it. Keeping the two tightly coupled works well, but it often provides more than you need. You can always throw compute at an index to make it fast, but it quickly adds up on your bill. As the cost grows, your data access patterns start to matter a lot, and the simple solution slowly turns into an operational overhead. I wanted to see if I could pull the storage and compute apart and build something that's cheap by default, so you only spend where it actually makes a difference. Many small-scale vector stores fit quite comfortably in memory and don't need additional storage. I wanted to check if a large vector index can live entirely in S3, with the machine in front of it deciding how many searches a second you get. I reconfigured demo.polign.com https://demo.polign.com , 12.5 million Wikipedia passages, into one S3 bucket as the database storage, 81.7 GiB in 142,653 objects, and tested it with several compute and memory profiles. tl;dr the premise of this exercise: - All of English Wikipedia 12.5 million passages is indexed once and sits in an S3 bucket. The storage cost is at $2.28 a month. - The compute primarily drives the search rate and usage cost. - Polign is compared to managed vector databases on cost and usage patterns, to get closer to understanding which one suits each use case. - Scaling up means launching a bigger box against the same bucket. The index stays where it is and the new box warms up from the first queries. When there is no traffic, stop the box and only pay for the bucket. - Accuracy remains consistent on every compute profile: Hit@10 0.47 on the same 10,000 NQ-Open questions on every box. The setup primarily affects the performance in terms of latency, QPS, and network/IO saturation. - Polign is cheapest at medium and high traffic. A $55 spot box handles a dozen searches a second for about the same as the cheapest managed vendors and 100x less than the metered ones. A $97 spot box handles hundreds a second, 2.26 billion searches a month at 4 cents per million, 25x less than the cheapest vendor. For a few searches a minute it is not cheaper, there is just no minimum bill. Compute selection based on your traffic The important part of this exercise is not the Wikipedia indexing. The index and storage never changes. It isn't copied, rebuilt or resized. Only the machine serving it changes to determine how aggressively the storage needs to be served. Sporadic traffic: a few searches a minute - This setup is suitable for most of individual workloads, side projects, internal search tools and access to the agent memory. - The cheapest box that is capable of handling this traffic is a t4g.small at $10.99 a month on spot - The search QPS is at about 6 searches a second sustained and bursts to 30. - The benchmark runs at 100k searches a month, which is about 3,300 searches a day. - When there is no traffic, the bucket remains parked incurring $2.28 a month. | At ~100k searches a month | Monthly | How it is priced | |---|---|---| | polign db, t4g.small on spot | $11.47 | $10.99 node + bucket, $0.48 of S3 requests | | polign db, parked bucket, node on demand | $2.28 + minutes of compute | the bucket alone; a node only while it answers | | Zilliz Cloud serverless https://zilliz.com/pricing | ~$16 | 100k × ~38 vCU × $4 per million vCU = $15, plus $1.10 storage | | turbopuffer https://turbopuffer.com/pricing | $16 | $16 floor; usage would be 100k × 1.28 GB × $1 per PB = $0.13, plus $0.60 storage | | Weaviate Cloud Flex https://weaviate.io/pricing | $45 | $45 floor; usage is 4.81B dims × $0.00465 per million = $22 | | Pinecone Standard https://www.pinecone.io/pricing/ | ~$54 | 100k × 28 read units × $16 per million = $45, plus $9.30 storage; the $50 floor does not bite | | Qdrant Cloud https://qdrant.tech/pricing/ | ~$120-200 | a ~8 GB RAM cluster from their calculator, running all month | Two of the serverless vendors are within a few dollars of a small box here, and their parked data is instantly queryable. Ours is not. At this traffic level polign db is not cheaper. What you get instead is no minimum bill, a bucket in your own AWS account, and the same bucket serving every setup below without moving the data. Moderate traffic: 10 to 20 searches a second - The benchmark runs at 30 to 50 million searches a month. - The traffic tested on 4 vCPU and 8 GiB instance. | At ~50M searches a month | Monthly | How it is priced | |---|---|---| | polign db, c7gd.xlarge on spot | $56 | $55.01 node + bucket, ~$1 of S3 requests | | turbopuffer https://turbopuffer.com/pricing | ~$65 | 50M × 1.28 GB minimum scan × $1 per PB = $64, plus $0.60 storage | | Weaviate Cloud Flex https://weaviate.io/pricing | $45 | $45 floor covers the usage; whether the Flex tier sustains 19 a second is not published | | Qdrant Cloud https://qdrant.tech/pricing/ | ~$120-200 | a cluster sized for the rate, from their calculator | | Zilliz Cloud serverless https://zilliz.com/pricing | ~$7,600 | 50M × ~38 vCU × $4 per million vCU = $7,600 | | Pinecone Standard https://www.pinecone.io/pricing/ | ~$22,400 | 50M × 28 read units × $16 per million = $22,400 | Scaling traffic: hundreds of searches a second - The r7gd.xlarge with four cores and 32 GiB of RAM, running $96.76 a month on spot - The test held 861 searches a second for ten minutes. - The benchmark capable of running 2.26 billion searches a month, which is about 75 million searches a day. - This is a spot price. If spot reclaims the r7gd, the NVMe cache goes with it. The replacement warms from the bucket, and in my run that first ten minutes from an empty disk still averaged 738 searches a second at concurrency 8, with the page cache full after a couple of minutes. | At ~2B searches a month | Monthly | How it is priced | |---|---|---| | polign db, r7gd.xlarge on spot | $97 | $96.76 node + bucket, $0 of S3 requests | | turbopuffer https://turbopuffer.com/pricing | ~$2,560 | 2B × 1.28 GB minimum scan × $1 per PB = $2,560 | | Weaviate Cloud https://weaviate.io/pricing , Qdrant Cloud https://qdrant.tech/pricing/ | not at their entry tiers | a cluster large enough for 860 a second, priced by quote | | Zilliz Cloud serverless https://zilliz.com/pricing | ~$304,000 | 2B × ~38 vCU × $4 per million vCU = $304,000 | | Pinecone Standard https://www.pinecone.io/pricing/ | ~$896,000 | 2B × 28 read units × $16 per million = $896,000 | At this rate the cheapest managed option costs 25x a spot box, and the metered ones cost more than most companies' whole cloud bill. This is what the architecture is built for. The corpus size is not in the formula. A search reads the cells it probes, the box keeps them in memory, and you pay for the box. Benchmark results - Four instance types, same index, same 10,000 NQ-Open questions, nprobe=4, so each search reads four index cells. - Each run held for ten minutes. Short runs ride the EC2 burst allowance and overstate the rate, see the Burst vs sustained tab. - Cost is the full monthly bill at on-demand prices, with spot underneath: the instance, a 20 GiB root volume and the index in S3. - No provisioned EBS volumes anywhere. I tried one in an earlier pass and it cost more than the instance it was attached to. Same index, four boxes Rows are ordered by cost. Filled dots read straight from S3; rings read through a warm local cache on the instance's NVMe. † The t4g.small could not hold its rate cleanly: 1.4% of its requests timed out over the ten minutes. Every other run had zero errors. Throughput and latency | Configuration | Searches / sec | p50 | p99 | Hit@10 | Peak RSS | CPU | |---|---|---|---|---|---|---| | t4g.small, S3 2 vCPU / 2 GiB | 6.6† | 213 ms | 7,799 ms | 0.470 | 1.0 GiB | 29% | | c7gd.xlarge, NVMe cache 4 vCPU / 8 GiB, 237 GB NVMe | 20.7 | 690 ms | 2,548 ms | 0.472 | 1.8 GiB | 4% | | r7gd.xlarge, NVMe cache 4 vCPU / 32 GiB, 237 GB NVMe | 861.4 | 35 ms | 92 ms | 0.470 | 6.8 GiB | 91% | Concurrency 4 / 8 / 16 / 32 in row order. † is the t4g.small's 1.4% of timeouts, see the note under the chart. Hit@10 is on the 10,000-question load set and is the same in every row: the box changes how fast you get the answer, not what it is. Latency is the full round trip from a separate load generator through the HTTP endpoint, including the full JSON response with passage text. Query embedding runs on the client and is not included. Peak RSS is the process's peak memory, which on a Go server is basically the budget you give it. CPU is the process's share of the whole host. - The CPU column tells you what each box is waiting on. The two boxes reading straight from S3 stay under 30%. They are waiting on the network. The c7gd at 4% is waiting on its disk. The r7gd at 91% is the only one actually using its CPU, and it is the one doing 860 a second. - On the S3 path, the query rate is the network allowance divided by bytes per query. A search at nprobe=4 reads about 17 MiB of index cells. The c7g.xlarge's 1.875 Gb/s baseline divided by 17 MiB is 13 a second, and it measured 12.7. At nprobe=1 6 MiB per query and concurrency 32 the same box did 211.7 a second. CPU is never the limit on this path. Cost | Configuration | Fixed per month | Searches per month | S3 requests that month | All in per month | Cost per request | |---|---|---|---|---|---| | t4g.small, S3 | $18.80 $14.60 + $1.92 + S3 spot $10.99 | 17.4M† 6.6 / sec | $84 $4.82 per 1M | $103 spot $95 | $5.90 per 1M spot $5.45 | | c7gd.xlarge, local cache | $169.76 $165.56 + $1.92 + S3 spot $55.01 | 54.4M 20.7 / sec | $1 $0.02 per 1M | $171 spot $56 | $3.14 per 1M spot $1.03 | | r7gd.xlarge, local cache | $227.73 $223.53 + $1.92 + S3 spot $96.76 | 2.26B 861.4 / sec | $0 $0.00 per 1M | $228 spot $97 | $0.101 per 1M spot $0.043 | Monthly, us-west-1, September 2026 prices, 730 hours, on-demand with spot underneath. Fixed is the instance, a 20 GiB gp3 root volume $1.92 and the index in S3 $2.28 . The NVMe boxes' disks are part of the instance price. Searches per month is the held rate times 730 hours, so what the box can do flat out. The requests column is what those searches cost in S3 GETs: 10 to 12 per search on the S3 path and near zero through the cache. Same-region S3 transfer is free. No public IP in any row. Spot prices are us-west-1b on the day. - On the S3 path, requests cost as much as the box. At 10 to 12 GETs per search, the c7g.xlarge spends $141 a month on requests against $135 for the instance. - A local disk takes requests out of the bill. The c7gd is the same price as the c7g on spot, and its request line drops from $141 to $1. That is the difference between $5.88 and $1.03 per million searches. - RAM takes the disk out of the path. The r7gd costs $42 a month more than the c7gd on spot and does 40x the searches, at 4 cents per million. Burst vs sustained EC2 instances have a baseline network allowance and a larger burst allowance that lasts a few minutes. A 60-second run makes a box reading from S3 look about 3x better than it is. c7g.xlarge at concurrency 8: | nprobe | 60-second run | Ten-minute run | p50, ten min | p99, ten min | Hit@10 | |---|---|---|---|---|---| | 8 | 20.3 / sec | 6.0 / sec | 1,078 ms | 5,576 ms | 0.516 | | 4 | 43.7 / sec | 12.7 / sec | 494 ms | 2,742 ms | 0.475 | | 2 | 59.0 / sec | 26.8 / sec | 226 ms | 1,458 ms | 0.420 | | 1 | 74.3 / sec | | | | 0.360 | Short runs use the 150 ms hedging default and a 256 MiB segment cache. Ten-minute runs use hedging off and a 2 GiB segment cache, the better settings, and still come in lower because the burst allowance is used up. Every number in the scenarios above is a ten-minute number. c7gd.xlarge: the disk is the limit - An xlarge gets one eighth of a physical drive. That slice gave about 300 MB/s at 100% utilization, 11 ms per read, with the CPU at 95% iowait. - The working set these queries touch is around 20 GB. An 8 GiB box cannot hold that in page cache. Even with the RAM given to the kernel instead of the Go heap a 2 GiB limit and a 512 MiB segment cache, which is what its row uses , three reads out of four still go to the device. - Concurrency 16 gives the same rate as 8 with double the p99. r7gd.xlarge: the CPU is the limit - Four times the memory of the c7gd, same disk. The working set fits in page cache. - The warm-up pass alone, ten minutes at concurrency 8 from an empty disk, averaged 738 a second. The burst allowance filled the page cache in the first couple of minutes. - Once warm, the rate is flat from concurrency 8 to 32, so the four cores are the limit. Run it at 8: same rate, a third of the p99. | nprobe | Concurrency | Run | Searches / sec | p50 | p99 | |---|---|---|---|---|---| | 4 | 8 | 5 min | 865.3 | 7.8 ms | 28 ms | | 4 | 16 | 5 min | 855.9 | 16.8 ms | 51 ms | | 4 | 32 | 10 min | 861.4 | 34.5 ms | 92 ms | | 8 | 32 | 5 min | 452.2 | 66.8 ms | 170 ms | Zero errors in every run, 0.00 S3 requests per search, peak RSS 6.9 GiB, CPU 91% of the host, about 1 MiB/s read from the NVMe. In an earlier pass a 16 vCPU c7g.4xlarge with a provisioned 1,000 MiB/s EBS volume reached 2,033 a second, but at $296 a month on spot, $131 of it for the volume, it costs more per search than this box. So it is not in the table. Try it The demo is live at demo.polign.com https://demo.polign.com , running on the t4g.small from the first scenario. To run this on your own bucket, start with the getting started guide developers.html .