A LlamaIndex vector store is where your index keeps node embeddings, usually alongside the text chunks and metadata, so queries can find similar content. The default store lives in memory, so a restart or redeploy wipes the index and sends you back to your embedding provider to pay for the same corpus again. Production systems move to a persistent backend such as TiDB, which adds SQL, metadata filtering, and distributed scale.
This tutorial walks the build path from document to a first query, using the LlamaIndex vector store documentation and the TiDB integration guide as references. SQL matters here because retrieval rarely stays a pure similarity problem. You filter by tenant, source, or date, and you want those fields in the same system as the vectors. By the end you will have a working TiDBVectorStore, a metadata-filtered query, and a checklist for taking it to production.
Key takeaways
- The default LlamaIndex vector store lives in memory, so a restart wipes the index unless you persist it yourself.
- Swapping in TiDB takes one StorageContext argument. The VectorStoreIndex code you already wrote stays the same.
- Metadata filters narrow nearest-neighbor results by fields such as tenant or source, which keeps multi-user retrieval scoped correctly.
- TiDBVectorStore supports only LlamaIndex’s default query mode, so keyword and hybrid retrieval need a separate SQL or pytidb path.
- TiDB vector search runs onTiDB Cloud Starter orTiDB Self-Managed v8.4.0 and later (v8.5.0 or later recommended).
How to Build a LlamaIndex Vector Store #
You build a LlamaIndex vector store on TiDB in five steps: load documents, split them into nodes, create a TiDBVectorStore, wrap it in a StorageContext, and build a VectorStoreIndex from it. The index code matches the in-memory version. Only the storage context changes.
The table below maps each piece of that flow to what LlamaIndex does by default and what a production setup replaces it with.
| Component | Default behavior | Production alternative | TiDB role |
|---|---|---|---|
| Vector storage | In-memory store inside your Python process | Database-backed vector store | TiDBVectorStore writes embeddings to a TiDB table |
| Persistence | Lost on exit unless you persist to local disk | Durable remote table | Data survives restarts and redeploys, and you can inspect it with SQL |
| Metadata filters | Applied to in-process data | Applied by the backend during nearest-neighbor search | TiDBVectorStore accepts LlamaIndex MetadataFilters |
| Multi-user access | One process owns the index | Shared store reached over the network | Many app instances connect to one cluster with a connection string |
| Scale | Bounded by one machine’s RAM | Store that scales out | TiDB distributes rows across TiKV nodes, and vector indexes run on TiFlash replicas |
You need three things before you start: Python 3.10 or later, an OpenAI API key for embeddings, and a TiDB cluster. The TiDB docs recommend a TiDB Cloud Starter instance. TiDB Self-Managed also works on v8.4.0 or later, with v8.5.0 or later recommended. Install the packages:
pip install llama-index llama-index-vector-stores-tidbvector
Load Documents and Create Nodes
LlamaIndex reads files into Document objects, then splits them into nodes, which are the chunks that get embedded and retrieved. Attach metadata at this stage, because every node inherits its parent document’s metadata. That is what makes tenant or document-type filters possible later.
import getpass
import os
from llama_index.core import Settings, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API key: ")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
documents = SimpleDirectoryReader("./data/docs").load_data()
for document in documents:
document.metadata["tenant"] = "acme"
document.metadata["doc_type"] = "runbook"
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = splitter.get_nodes_from_documents(documents)
print(f"{len(documents)} documents became {len(nodes)} nodes")
Connect StorageContext to TiDBVectorStore
Create the vector store with your connection string, then hand it to a StorageContext. The vector_dimension value must match your embedding model. TiDB vector search accepts vectors of up to 16,383 dimensions, so common embedding models fit. Copy the PyMySQL connection string from the Connect dialog of your TiDB Cloud Starter instance.
import getpass
from llama_index.core import StorageContext
from llama_index.vector_stores.tidbvector import TiDBVectorStore
tidb_connection_string = getpass.getpass("TiDB connection string: ")
vector_store = TiDBVectorStore(
connection_string=tidb_connection_string,
table_name="docs_vectors",
distance_strategy="cosine",
vector_dimension=1536,
drop_existing_table=False,
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
Keep drop_existing_table=False outside of throwaway experiments. Setting it to True drops the table on startup.
Index Data and Run the First Query
Pass the nodes and the storage context to VectorStoreIndex. LlamaIndex generates embeddings and writes them to the docs_vectors table. From then on, any process with the connection string can reopen the same index without re-embedding anything.
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex(nodes, storage_context=storage_context, show_progress=True)
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("How do we rotate the API keys?")
print(response)
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
The TiDB docs example uses VectorStoreIndex.from_documents(...), which runs the same load-split-embed-store steps in one call. Splitting nodes yourself, as above, gives you control over chunk size and overlap. For a longer end-to-end example with a local model, see this LlamaIndex RAG implementation guide.
What Does a LlamaIndex Vector Store Actually Store? #
A LlamaIndex vector store holds one embedding per node, and depending on the backend it can also hold the node text and its metadata. It answers one question: which stored vectors sit closest to this query vector? Everything else, including how you chunk, which filters you apply, and how many results you pass to the LLM, is retrieval policy that you design around it.
Documents, Nodes, and Embeddings
Four terms cause most of the confusion, so this article uses them the same way throughout.
| Term | What it is | Created by |
|---|---|---|
| Document | A source item as loaded, such as a file or a web page | A reader like SimpleDirectoryReader |
| Node | A chunk of a document that carries metadata and a link back to its source | A node parser like SentenceSplitter |
| Embedding | The vector that represents one node’s text | Your embedding model |
| Metadata | Key-value fields attached to a document or node | You, at load time |
The vector store persists embeddings for nodes. It does not persist your original files, so keep those in object storage or wherever they live today.
Metadata and Filterable Fields
Metadata is how you carry business context into retrieval. Fields such as tenant, source, doc_type, or published_at let a query say “only nearest neighbors from this customer’s runbooks.” With TiDBVectorStore, the table holds the vector, the node text, and a JSON metadata column, and you can confirm the layout with DESCRIBE docs_vectors in any MySQL-compatible client.
By default, LlamaIndex also includes metadata in the text it embeds and in the context it sends to the LLM. If a field exists only for filtering, such as an internal tenant ID, exclude it from both by setting excluded_embed_metadata_keys and excluded_llm_metadata_keys on the document.
Retrieval Context for RAG and Agents
For retrieval-augmented generation, the retrieved nodes become the context in the LLM prompt, so the store’s job is to return the right few chunks. For agents with long-term memory, the same mechanism stores prior conversation turns or extracted facts as nodes, then retrieves them by similarity at the start of a later session. In both cases, storage, indexing, and retrieval policy are separate decisions. A database gives you durable storage and filtering. Chunking, top-k, and filter rules remain your code.
When Should You Move From the In-Memory Store to TiDB? #
Move to a persistent store when the index has to outlive one process, serve more than one user, or hold data you cannot afford to re-embed after every deploy. The default in-memory store is the right tool until one of those conditions shows up. Re-embedding is the hidden cost: every restart means paying your embedding provider again for the same corpus and waiting for it to finish.
Prototype Fast with the Default Store
For a notebook, a demo, or a corpus that fits comfortably in RAM, the default needs no setup. You can even write it to disk and reload it, which covers a single-developer workflow.
from llama_index.core import StorageContext, VectorStoreIndex, load_index_from_storage
index = VectorStoreIndex(nodes) # in-memory vector store
index.storage_context.persist(persist_dir="./storage")
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="./storage"))
Add Persistence for Production Retrieval
Local persistence breaks down when the app runs on more than one machine or the corpus keeps growing. Common triggers are:
- A corpus too large for one machine’s memory.
- Several app replicas that need the same index.
- Per-tenant or per-source filtering.
- Agents that must remember across sessions.
- Deployment pipelines where the container filesystem is disposable.
Here is an illustrative example. A support-docs assistant starts as a script over 200 files on a laptop. Six months later it serves 40 customers from three API replicas, each customer sees only its own docs, and new content arrives daily. The in-memory index now needs a shared home, a tenant filter, and incremental inserts. Moving to TiDBVectorStore covers all three without changing the query code, because the replicas call VectorStoreIndex.from_vector_store(...) against one table.
Support Long-Term Memory for AI Agents
An agent that remembers earlier sessions needs the same properties: durable storage, filtering by user, and retrieval by meaning. A simple pattern writes each memory worth keeping, such as a stated preference or a resolved issue, as a node with user_id and kind metadata. At the start of a new session, you query with a filter on that user. For a broader look at what an agent memory database needs, see this guide to choosing a database for AI agents. PingCAP also describes a persistent context pattern for AI agents.
from llama_index.core.schema import TextNode
memory = TextNode(
text="User wants all code samples in Python and prefers short answers.",
metadata={"user_id": "u_123", "kind": "memory", "session_id": "s_42"},
)
index.insert_nodes([memory])
What TiDB Adds
TiDB combines SQL, vector search, JSON metadata, and horizontal scale in one MySQL-compatible distributed SQL database, which makes it a distributed vector database for RAG without a separate store for each retrieval concern. The TiDB integration overview lists LlamaIndex as an officially supported AI framework, and the LlamaIndex integration guide covers TiDB Cloud Starter and TiDB Self-Managed v8.4.0 or later, with v8.5.0 or later recommended. The same docs mark vector search as a public preview feature, so check the release notes before you plan a launch around it. For more patterns on the same foundation, browse the guide to build AI applications with vector search.
Keep Operational Data and Vectors Together
MySQL compatibility matters here because your application likely already has users, tenants, permissions, and content tables behind MySQL drivers and ORMs. With TiDB, the vector table lives in the same cluster, so you can inspect it with the SQL tools you already use and relate retrieved nodes to business records by key. TiDB shards data across TiKV nodes by key range without application-level sharding, which keeps the same schema working as the corpus grows.
Filter and Retrieve in One Backend
TiDBVectorStore translates LlamaIndex metadata filters into a SQL WHERE clause over the table’s JSON metadata column, so tenant, source, and document-type constraints are enforced in the database. By default, the filter runs before the distance ordering. According to the TiDB docs, that pattern returns exact results but cannot use a vector index, so cost grows with the number of rows that pass the filter. Metadata filtering is a retrieval-scoping tool, and your application’s authorization layer still decides who may ask for what.
TiDB is not the only reasonable backend. If your corpus is small, your team already runs a single PostgreSQL instance, and you have no plans for multi-tenant scale, adding another database may not pay off. TiDB earns its place when you want SQL, vectors, and horizontal scale in one system and your team already works with MySQL tooling.
How Do Metadata Filters and Keyword Search Improve LlamaIndex Retrieval? #
Embeddings rank by meaning, which is the wrong tool when the right answer depends on who owns the data, which document type it came from, or an exact identifier such as an error code. Metadata filters restrict results to rows that match fields such as tenant or document type. Keyword search handles identifiers. Semantic search alone is enough when users ask open-ended questions over a uniform corpus.
Filter by Tenant, Source, and Document Type
LlamaIndex expresses filters with MetadataFilters. Multiple filters combine with AND by default.
from llama_index.core.vector_stores.types import MetadataFilter, MetadataFilters
filters = MetadataFilters(
filters=[
MetadataFilter(key="tenant", value="acme", operator="=="),
MetadataFilter(key="doc_type", value="runbook", operator="=="),
]
)
query_engine = index.as_query_engine(filters=filters, similarity_top_k=3)
response = query_engine.query("How do we rotate the API keys?")
The TiDB docs show how strict this is. With a book != "paul_graham" filter on an index where every node has book = "paul_graham", the query returns Empty Response instead of the closest semantic match. An empty result is the correct outcome when nothing passes the filter, so handle it explicitly in your app.
Add Keyword Search for Identifiers and Error Codes
TiDBVectorStore supports only LlamaIndex’s default query mode, so LlamaIndex-native hybrid mode is not available on this backend. Keyword retrieval has to run as its own path. TiDB’s full-text search provides one: you add a full-text index to a text column and rank matches by BM25 relevance with fts_match_word(). The node text lives in the document column of your vector table.
ALTER TABLE docs_vectors
ADD FULLTEXT INDEX (document) WITH PARSER MULTILINGUAL ADD_COLUMNAR_REPLICA_ON_DEMAND;
SELECT id, document FROM docs_vectors
WHERE fts_match_word("ERR_4012", document)
ORDER BY fts_match_word("ERR_4012", document) DESC
LIMIT 10;
fts_match_word() tokenizes the query and matches any token, so it does not do exact phrase matching. Test how your parser tokenizes identifiers such as ERR_4012 before you rely on it for error-code lookups.
Full-text search is still early. It is currently available only on TiDB Cloud Starter in five AWS regions, so check the full-text search docs for the current list before you design around it. The hybrid search guide shows how the pytidb SDK fuses vector and keyword results with reciprocal rank fusion or weighted scores.
To decide whether you need keyword search, run this comparison on your own data:
- Collect 20 real queries, including some with exact identifiers such as error codes or product SKUs.
- Run each query three ways: keyword search only, semantic search with metadata filters, and semantic search without filters.
- Record whether the correct chunk appears in the top five for each method.
- Add a keyword path only if identifier queries fail under semantic ranking.
Reduce Hallucinations With Better Retrieval Context
An LLM that receives the wrong chunks will answer from them anyway. Filters remove chunks from the wrong tenant or document type, and keyword matching rescues identifier queries that embeddings blur. Neither guarantees a correct answer, so measure retrieval quality directly instead of judging it from final responses. For questions that depend on relationships between entities, GraphRAG with a knowledge graph can add context that chunk similarity misses.
Checklist for a Production-Ready Vector Store #
A production-ready LlamaIndex vector store has deliberate chunking, a pinned embedding model, planned metadata, tuned inserts, measured retrieval, enforced tenant isolation, and logging. Use this list before you ship.
- Chunking: Choose chunk size and overlap on purpose. This tutorial uses 512 and 50. Re-test when your document types change.
- Embedding model: Pin the model and
vector_dimensiontogether. Switching models means re-embedding the corpus into a new table. - Metadata design: Decide filterable fields up front, such as tenant, source, doc type, and timestamp. Keep values consistent, and exclude filter-only fields from embeddings.
- Insert batch size: Tune it for remote backends and larger loads (details below).
- Vector index: With
tidb-vector0.0.15, TiDBVectorStore does not create a vector index. For large tables, add an HNSW index yourself, which needs a TiFlash replica. Then runEXPLAINon a filtered query and look forannIndex:. Filters applied before the distance ordering return exact results but scan every matching row, so cost grows with the filtered set. See the TiDBvector search index docs. - Retrieval evaluation: Keep a fixed set of queries with known correct chunks, and track top-k hit rate before and after each change.
- Multi-user isolation: Apply the tenant filter on every query in application code. If you need a hard boundary, give each tenant its own table. Partitioned tables cannot have vector indexes, so partitioning by tenant means queries run without one.
- Deletes and updates: Plan how removed or changed source documents leave the index.
vector_store.delete(doc_id)removes a document’s nodes. - Secrets: Read the connection string from a secret manager or environment variable, not a prompt.
- Observability: Log each query’s filters, returned node IDs, scores, and latency. The TiDBperformance tuning docs cover what to check on large tables.
On batch size: the LlamaIndex TiDB example notes that a smaller insert_batch_size, such as 1000, can avoid exceeding the packet size limit during insertion. Remote databases and large vector loads are where this tends to matter, so lower it if inserts fail with packet-size errors.
index = VectorStoreIndex(
nodes,
storage_context=storage_context,
insert_batch_size=1000,
show_progress=True,
)
If you are still choosing a backend, this comparison of the best vector databases for LlamaIndex covers the trade-offs.
Sources checked October 30, 2026, with llama-index 0.14.25 and llama-index-vector-stores-tidbvector 0.6.0: the LlamaIndex vector store docs*, the* LlamaIndex TiDB vector store page*, and the* TiDB integration guide*.*
Build LlamaIndex Retrieval on TiDB #
A LlamaIndex vector store on TiDB keeps embeddings, node text, and metadata in one table that survives restarts and that every app replica can query with the same filters.
To try it on your own documents, create a free TiDB Cloud Starter instance, paste its PyMySQL connection string into the TiDBVectorStore call above, and measure top-k hit rate with the 20-query comparison from the filtering section. If you plan to add keyword search, create the instance in one of the AWS regions where full-text search is available.
LlamaIndex Vector Store FAQs #
What Is a LlamaIndex Vector Store?
A LlamaIndex vector store is the storage layer behind a VectorStoreIndex. It keeps embeddings for your nodes and, depending on the backend, the node text and metadata. When you query, LlamaIndex embeds the question, asks the vector store for the nearest nodes, and passes them to the LLM. You choose the backend through a StorageContext, so the same index code works with an in-memory store or a database such as TiDB.
Is the Default LlamaIndex Vector Store Persistent?
No. By default, LlamaIndex keeps vectors in an in-memory store inside your Python process, so they disappear when it exits. You can write the index to local disk with storage_context.persist(), which works for single-machine prototypes. Once several services share the index, the corpus outgrows memory, or deployments replace your filesystem, use a database-backed store instead.
Can I Use a SQL Database as a LlamaIndex Vector Store?
Yes. LlamaIndex supports database-backed vector stores, including TiDB through the llama-index-vector-stores-tidbvector package. TiDB is a MySQL-compatible distributed SQL database with built-in vector search, so one system holds vectors, metadata, and application data. Teams choose a SQL backend for durability, shared access, and filtering with tools they already know. Because TiDBVectorStore supports only LlamaIndex’s default query mode, plan a separate SQL path if you need keyword search.
How Does a Vector Store Support Long-Term Memory for AI Agents?
Store each memory worth keeping, such as a preference, a decision, or a conversation summary, as a node with metadata like user_id and session_id. At the start of a new session, embed the user’s message and retrieve the closest memories, filtered to that user. Because the vector store is persistent, memories survive restarts and can be shared across agent instances.
Do I Need Metadata Filters for LlamaIndex Retrieval?
You need them whenever relevance depends on more than meaning, such as tenant boundaries, document types, sources, or dates. Without filters, a similarity search can return a near match from the wrong customer or an outdated document. Filters improve precision, but they are not a security control on their own, so keep authorization checks in your application. A single-user prototype over a uniform corpus can skip them.
Experience modern data infrastructure firsthand.
TiDB Cloud Dedicated #
A fully-managed cloud DBaaS for predictable workloads
TiDB Cloud Starter #
A fully-managed cloud DBaaS for auto-scaling workloads