{"slug": "show-hn-glider-a-single-node-vector-database-with-s3-as-its-only-durable-state", "title": "Show HN: Glider: A single-node vector database with S3 as its only durable state", "summary": "Developer Omer Feyzioglu released Glider, a single-node vector database that treats S3-compatible object storage as its only durable state, acknowledging each write only after it is persisted to the bucket. Glider serves nearest-neighbor search with metadata filters over an HTTP/JSON API and ships a dependency-free Python client plus an MCP memory server for AI agents, with optional server-side embedding via a local 384-dimension ONNX model, Ollama, or OpenAI's text-embedding-3-small. Because the bucket is the database, a container can be stopped and restarted anywhere with the same S3 settings and credentials and retain its data, with local RAM and SSD used only to speed reads.", "body_md": "**Vector search with S3 as the source of truth.**\n\n[Website and interactive storage simulation](https://gliderdb.com/)\n\nGlider is a single-node vector database. Every acknowledged write is durable in S3-compatible object storage before the server answers; local RAM and SSD only speed up reads and never hold the only copy. It serves nearest-neighbor search with metadata filters over an HTTP/JSON API, with a Python client and an MCP memory server for AI agents.\n\nStart a server with a local directory as storage (data is lost when the container stops):\n\n```\ndocker run --rm -p 8080:8080 \\\n  -e GLIDER_DATA_DIR=/var/lib/glider/data ghcr.io/omerfeyzioglu/glider:latest\n```\n\nCreate a collection, write two points and query them:\n\n```\ncurl -sS localhost:8080/v1/collections -H 'content-type: application/json' \\\n  -d '{\"name\":\"demo\",\"dimensions\":3}'\n\ncurl -sS localhost:8080/v1/collections/demo/write -H 'content-type: application/json' \\\n  -d '{\"upsert\":[{\"id\":1,\"vector\":[0,0,0],\"metadata\":{\"color\":\"red\"}},{\"id\":2,\"vector\":[1,1,1]}]}'\n\ncurl -sS localhost:8080/v1/collections/demo/query -H 'content-type: application/json' \\\n  -d '{\"vector\":[1,1,0.9],\"k\":2,\"include_metadata\":true}'\n```\n\nThe same with the dependency-free Python client:\n\n```\n# pip install \"git+https://github.com/omerfeyzioglu/glider#subdirectory=clients/python\"\nfrom glider_client import Client\n\nclient = Client(\"http://localhost:8080\")\nclient.create_collection(\"docs\", dimensions=3, metric=\"cosine\")\ndocs = client.collection(\"docs\")\ndocs.upsert([{\"id\": 1, \"vector\": [0, 0, 1], \"metadata\": {\"lang\": \"en\"}}])\nprint(docs.query([0, 0, 1], k=1, include_metadata=True))\n```\n\nOpen [http://localhost:8080/console](http://localhost:8080/console) to browse collections and run queries\nin the browser.\n\nFor S3 or MinIO storage, the Docker Compose demo and building from source,\nsee [installation](https://github.com/omerfeyzioglu/glider/blob/main/docs/INSTALL.md); every setting is listed in\n[configuration](https://github.com/omerfeyzioglu/glider/blob/main/docs/CONFIGURATION.md).\n\nWith a bucket and AWS credentials exported in your shell:\n\n```\ndocker run --rm -p 8080:8080 -v glider-cache:/var/lib/glider/cache \\\n  -e GLIDER_S3_BUCKET=my-bucket -e GLIDER_S3_NAMESPACE=glider \\\n  -e GLIDER_S3_REGION=eu-central-1 \\\n  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \\\n  ghcr.io/omerfeyzioglu/glider:latest\n```\n\nNothing else is needed: the bucket is the database; stop the container and start\na new one anywhere with the same settings and access, and the data is there.\nSee [Run on AWS](https://github.com/omerfeyzioglu/glider/blob/main/docs/INSTALL.md#run-on-aws) for setup and IAM, or\n[S3-compatible storage](https://github.com/omerfeyzioglu/glider/blob/main/docs/INSTALL.md#s3-compatible-storage) for MinIO.\n\nEnable server-side embedding to query with `{\"text\":\"a sleepy cat\"}` or use\nText mode in the console. Embed documents with `POST /v1/embed`, then write\nits vectors with optional `metadata.text`. Stored and acknowledged data stays\nvectors. The default local model needs a 384-dimension cosine collection.\n\n```\n# Local ONNX model; downloaded on first use into the persistent volume.\ndocker run --rm -p 8080:8080 -v glider-text:/var/lib/glider \\\n  -e GLIDER_DATA_DIR=/var/lib/glider/data -e GLIDER_EMBED_PROVIDER=local \\\n  ghcr.io/omerfeyzioglu/glider:latest\n\n# Ollama on the host (pull nomic-embed-text in Ollama first).\ndocker run --rm -p 8080:8080 -e GLIDER_DATA_DIR=/var/lib/glider/data \\\n  -e GLIDER_EMBED_PROVIDER=openai -e GLIDER_EMBED_MODEL=nomic-embed-text \\\n  -e GLIDER_EMBED_URL=http://host.docker.internal:11434/v1 \\\n  ghcr.io/omerfeyzioglu/glider:latest\n```\n\nFor OpenAI, set `GLIDER_EMBED_URL=https://api.openai.com/v1`, a model such\nas `text-embedding-3-small`, and `GLIDER_EMBED_API_KEY`. Use the response's\ndimensions when creating a collection. Embedding is off by default; source\nbuilds opt into `embed-local` and/or `embed-openai`. See\n[configuration](https://github.com/omerfeyzioglu/glider/blob/main/docs/CONFIGURATION.md#text-embedding-optional).\n\n- **Durable on S3.** Each write batch becomes one immutable log object, created\nconditionally before it is acknowledged; nothing is overwritten in place.\n- **Crash-safe takeover.** A restarted server waits out the old writer's\nlease, fences it at the object store and replays the log; no operator step\nis needed after a crash.\n- **Retry-safe writes.** Every write carries a request ID; resending it\nwithin the retained 128-commit window returns the original outcome\ninstead of applying the write twice. Supply and keep the ID before sending so a lost\nresponse can be retried ([retry contract](https://github.com/omerfeyzioglu/glider/blob/main/docs/API.md#request-ids-and-retries) ).\n- **Collections.** One server creates and serves many collections, each with\nits own dimension and metric (squared Euclidean, Manhattan or cosine).\n- **Metadata filters.** Equality, set, existence, numeric and logical\nfilters;`exact: true` answers any filter exhaustively.\n- **Clustered ANN that builds itself.** The server clusters a collection in\nthe background at 250,000 rows and rebuilds the clusters as it grows.\n- **SSD cache.** A local block cache warms in the background so warm queries\nneed no remote reads; losing it loses no data.\n- **Operations.** Prometheus metrics, a status endpoint, bearer-token\nauthentication, and`glider-admin` for backup and restore.\n\nThe Python client includes `glider-mcp`, an MCP server that gives agents\n(Claude Code, Claude Desktop, Cursor and other MCP clients) durable\n`remember`, `recall` and `forget` tools backed by Glider:\n\n```\ndocker run -d --name glider -p 8080:8080 -v glider-data:/var/lib/glider \\\n  -e GLIDER_DATA_DIR=/var/lib/glider/data ghcr.io/omerfeyzioglu/glider:latest\npip install \"glider-client[mcp] @ git+https://github.com/omerfeyzioglu/glider#subdirectory=clients/python\"\nclaude mcp add glider -e GLIDER_COLLECTION=memory -- glider-mcp\n```\n\nThe [client guide](https://github.com/omerfeyzioglu/glider/blob/main/clients/python/README.md#mcp-memory-for-agents) covers\nother MCP clients and settings.\n\n1,000,000 SIFT vectors (128 dimensions, k=10) in AWS S3 Standard, served from a c7g.2xlarge (8 vCPU, 16 GiB) in eu-central-1 while four writers and four readers run concurrently:\n\n| Measure | Result | \n|---|---|\n| Recall@10, static (mean / p5) | 0.998 / 1.0 | \n| Recall@10 after updates, cold cache (mean / p5) | 0.964 / 0.8 | \n| Query p95, warm cache | 29.8 ms | \n| Query p95, cold cache | 57.0 ms | \n| Write p95 | 86.9 ms | \n| Open / reopen | 2.93 / 3.84 s | \n\nThe run at revision `35f9b46` lost no acknowledged writes and returned equal\nresults after cache loss and backup restore. Methodology and raw results are in the\n[benchmark report](https://github.com/omerfeyzioglu/glider/blob/main/benchmarks/M39.md#current-main-on-s3) and\n[BENCHMARKS.md](https://github.com/omerfeyzioglu/glider/blob/main/BENCHMARKS.md).\n\nMany small collections on one server, same instance and S3 Standard:\n10,000 collections of 1,000 vectors (10,000,000 vectors) were created and\nloaded in 22 minutes with the server killed (`SIGKILL`) halfway; every\ntenant was verified with no lost or duplicated write. Warm queries over 64\nactive collections ran at 2,457 per second with a p95 of 18.5 ms and\nrecall@10 of 1.0; a cold tenant\nopened from S3 and answered its first query in 716 ms (p95). Idle\ncollections close, so they cost no S3 requests\n([details](https://github.com/omerfeyzioglu/glider/blob/main/BENCHMARKS.md#multi-tenant-server-scenario)).\n\n- **Write:** a single committer publishes each batch as one conditional\nlog object, then acknowledges it.\n- **Maintenance:** idle time seals the log into immutable packs and indexes,\nthen publishes a new root generation that names them.\n- **Read:** queries route to candidate blocks, read bounded byte ranges from\nS3 or the SSD cache, and rerank full vectors.\n\nThe [architecture guide](https://github.com/omerfeyzioglu/glider/blob/main/docs/ARCHITECTURE.md) adds an AWS deployment\npattern; [DESIGN.md](https://github.com/omerfeyzioglu/glider/blob/main/DESIGN.md) specifies formats, invariants and recovery.\n\n| Document | Contents | \n|---|---|\n| [Installation](https://github.com/omerfeyzioglu/glider/blob/main/docs/INSTALL.md) | Docker, Docker Compose with MinIO, building from source | \n| [Configuration](https://github.com/omerfeyzioglu/glider/blob/main/docs/CONFIGURATION.md) | Environment variables and serving profile | \n| [HTTP API](https://github.com/omerfeyzioglu/glider/blob/main/docs/API.md) | Endpoints, filters, errors, retries and consistency | \n| [Python client](https://github.com/omerfeyzioglu/glider/blob/main/clients/python/README.md) | Client API and MCP memory server | \n| [Operations](https://github.com/omerfeyzioglu/glider/blob/main/docs/SERVING.md) | Running, backup, restore, cache and clustering | \n| [Recovery](https://github.com/omerfeyzioglu/glider/blob/main/docs/RECOVERY.md) | Crash, takeover and restore procedures | \n| [Architecture](https://github.com/omerfeyzioglu/glider/blob/main/docs/ARCHITECTURE.md) | Runtime and AWS deployment diagrams | \n| [Design](https://github.com/omerfeyzioglu/glider/blob/main/DESIGN.md) | Formats, invariants and guarantees | \n| [Rust library](https://github.com/omerfeyzioglu/glider/blob/main/docs/LIBRARY.md) | Embedding the engine in Rust | \n| [Benchmarks](https://github.com/omerfeyzioglu/glider/blob/main/BENCHMARKS.md) | Measurements and how to reproduce them | \n| [Changelog](https://github.com/omerfeyzioglu/glider/blob/main/CHANGELOG.md) | Release notes | \n| [Contributing](https://github.com/omerfeyzioglu/glider/blob/main/CONTRIBUTING.md) | Development workflow and checks | \n| [Website](https://github.com/omerfeyzioglu/glider/blob/main/docs/WEBSITE.md) | Previewing the site and regenerating its playground results | \n| [Security](https://github.com/omerfeyzioglu/glider/blob/main/SECURITY.md) | Reporting vulnerabilities and deployment security | \n\n- Single node: no replication, sharding or standby; after a crash, the next server waits for the writer lease (10 s by default) before taking over.\n- Filtered queries are approximate and may return fewer than `k` results\nunless they use`exact: true` or the collection's resident filter.\n- Cache loss can lower approximate recall as well as increase latency;\nacknowledged data remains durable. Use `exact: true` for exhaustive results.\n- A collection's dimension and metric are fixed when it is created.\n- The server speaks plain HTTP; terminate TLS in a reverse proxy.\n- The measured 1M-vector collection opens from S3 in a few seconds; larger collections have no fixed open-time guarantee.\n\nContributions are welcome. See [CONTRIBUTING.md](https://github.com/omerfeyzioglu/glider/blob/main/CONTRIBUTING.md) for the\nworkflow and the checks CI runs.\n\nLicensed under either of [Apache License, Version 2.0](https://github.com/omerfeyzioglu/glider/blob/main/LICENSE-APACHE) or\n[MIT license](https://github.com/omerfeyzioglu/glider/blob/main/LICENSE-MIT) at your option. Unless you explicitly state\notherwise, any contribution intentionally submitted for inclusion in this\nproject, as defined in the Apache-2.0 license, shall be dual licensed as\nabove, without any additional terms or conditions.", "url": "https://wpnews.pro/news/show-hn-glider-a-single-node-vector-database-with-s3-as-its-only-durable-state", "canonical_source": "https://github.com/omerfeyzioglu/glider", "published_at": "2026-10-18 20:04:44+00:00", "updated_at": "2026-10-18 20:19:15.140879+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "agent-protocols", "ai-tools"], "entities": ["Glider", "Omer Feyzioglu", "Amazon S3", "MinIO", "Ollama", "OpenAI", "Model Context Protocol"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-glider-a-single-node-vector-database-with-s3-as-its-only-durable-state", "markdown": "https://wpnews.pro/news/show-hn-glider-a-single-node-vector-database-with-s3-as-its-only-durable-state.md", "text": "https://wpnews.pro/news/show-hn-glider-a-single-node-vector-database-with-s3-as-its-only-durable-state.txt", "jsonld": "https://wpnews.pro/news/show-hn-glider-a-single-node-vector-database-with-s3-as-its-only-durable-state.jsonld"}}