Ceph object storage can include a vector database and search to support AI large language model workloads.
The SNIA has produced a presentation slide deck discussing how Ceph can include vector search. For context, Ceph is massively scalable open-source object, file and block storage software. RADOS (Reliable Autonomous Distributed Object Store) is the object storage part with librados being the a client library providing low-level, direct access to RADOS. The RADOS Gateway (RGW), is an object storage interface built on top of the librados library to provide applications with a RESTful gateway to Ceph storage clusters.
There are so many acronyms used in this deck that we’ve included a mini glossary in the bootnote below.
Ceph can use Amazon’s S3 Vectors bucket type to store AI large language model (LLM) vectors and provide a vector search capability.The vectors are stored in LanceDB tables as dense float32 arrays with string metadata tags. Ceph’s S3 Vectors support approximate nearest neighbour (ANN) searches in the vector space, using query vectors with optional metadata tags to reduce the candidate vector set.
The Lance format stores both the data, and vector, full-text, and scalar indexes, all stored with the dataset rather than in an external service. Ceph remains the storage and control plane. Lance format provides vector-specific layout and retrieval mechanics. LanceDB provides a higher-level API for vector indexing, search and table operations. Lance format provides a lower-level API for the same functionality.
K-means clustering during index generation is accelerated via AVX2 SIMD. This minimally requires AVX2, which is relevant for modern x86 Ceph OSD nodes. There are tunable nprobes per query , with higher values increasing recall, and lower values reducing latency. The target latency range is 100–800 ms.
LanceDB supports L2, cosine, and dot for indexed vector search. Ceph's S3 Vectors path can focus on L2/cosine. L2 distance measures the straight-line distance between vector coordinates. Cosine distance measures the angle between vector coordinates from a start point.Both can rank nearest neighbours but they use different definitions of “nearest.”
When should you use S3 vectors in Ceph?
Cephs’ S3 Vectors are not a replacement for a dedicated vector database. If the workload requires single-digit millisecond latency, rich full-text plus vector hybrid search, or a complex query DSL, a dedicated system is still the right answer. S3 Vectors targets workloads where the requirement is cost-effective storage of large numbers of vectors with sub-second ANN search and simple metadata filtering — in a cluster that already runs Ceph, and doubly so if the data the embeddings have been extracted from what already exists in Ceph object storage.
You can find out more about S3 Vectors here and Ceph nearest neighbour search here. Bootnote
There are so many acronyms employed in this area we thought a mini-glossary would be helpful.
ANN - Approximate Nearest Neighbour; a type of search used in vector search operations. It finds the K most semantically similar items to a query vector.
AVX - a family of SIMD (Single Instruction, Multiple Data) instruction set extensions for x86 CPUs, allowing the processor to perform the same operation on multiple pieces of data in parallel using wide vector registers.
AVX2 - Intel’s Advanced Vector Extensions 2, extends earlier AVX with better integer support, fused multiply-add (FMA), gather instructions, and more.It’s available on x86 server CPUs.
float32 - float32, also called single-precision floating-point) is a common 32-bit data type used in computing to represent real numbers (decimals/fractions).
IVF - Inverted File.
IVF-PQ - IVF-PQ partitions the vector space into cells (IVF) and compresses vectors using product quantization (PQ). K-means clustering - LanceDB relies on an IVF-PQ index. K-means is the key algorithm that builds the IVF part of that index. It partitions the high-dimensional vector space into* K *clusters. Each vector is assigned to the nearest cluster centroid. (A centroid is the central point (or "average") of a group of data points in a cluster.) At query time, Ceph only searches the clusters closest to the query vector (controlled by nprobe), dramatically reducing the number of vectors that need to be compared.
Lance - a columnar data format used by Ceph.
LanceDB - a vector database/library built on top of the Lance format. It’s is a full-stack library built on the Lance columnar format, designed from the ground up for fast random-access disk I/O. It uses IVF-PQ indexes, supports both distance metrics, includes hybrid ANN + scalar predicate search, and provides a clean API whose dataset/table/index model maps directly onto S3 Vectors' bucket/index/vector abstractions. LanceDB accelerates K-means clustering during index generation using AVX2 SIMD. Apache 2.0 licensed.
nprobe - a query-time tuning parameter in IVF (Inverted File) indexes. During search, instead of comparing your query vector to every vector in the index (too slow), the system finds the nearest nprobe centroids to the query and only searches the vectors inside those nprobe clusters.
OSD - Ceph Object Storage Daemon.
OSD Node - A Ceph OSD node (also called an OSD host or storage node) is a physical or virtual server in a Ceph cluster that runs one or more OSD, and is the actual hardware/software unit where data is persistently stored.
PQ - Product Quantization.
RADOS - Ceph's Reliable Autonomous Distributed Object Store.
RGW - Ceph’s RADOS GateWay.
S3 Vectors - Amazon S3 Vectors is a purpose-built, cost-optimized feature of Amazon S3 object storage that adds native support for storing and querying large-scale vector data (embeddings) directly in S3. It is said to obviate the need for a separate vector database.
Scalar indices - Scalar indices make pre-filtering or inline filtering efficient by organizing scalar data (numbers, strings, booleans, dates, etc.) for fast lookups, ranges, equality checks, etc. Imagine search terms with a numeric limit, such as "only documents from 2025."
SIMD - Single instruction, Multiple Data lets an x86 CPU perform the same operation on multiple data elements in parallel (e.g., 8× 32-bit integers or 4× 64-bit integers at once in 256-bit registers).
Vector - An array of float32 numbers encoding the semantic meaning of any object - text, image, audio - as a point in a high-dimensional space.