How to Build a LlamaIndex Vector Store With TiDB

A LlamaIndex tutorial details how to replace the framework's default in-memory vector store with a persistent TiDBVectorStore, requiring only a change to the StorageContext argument while leaving existing VectorStoreIndex code unchanged. The guide, referencing LlamaIndex and PingCAP TiDB documentation, notes that TiDB vector search runs on TiDB Cloud Starter or TiDB Self-Managed v8.4.0 and later, with v8.5.0 or later recommended, and that TiDBVectorStore supports only LlamaIndex's default query mode, so keyword and hybrid retrieval need a separate SQL or pytidb path. The setup requires Python 3.10 or later, an OpenAI API key for embeddings, and installation of the llama-index and llama-index-vector-stores-tidbvector packages.

A LlamaIndex vector store is where your index keeps node embeddings, usually alongside the text chunks and metadata, so queries can find similar content. The default store lives in memory, so a restart or redeploy wipes the index and sends you back to your embedding provider to pay for the same corpus again. Production systems move to a persistent backend such as TiDB https://www.pingcap.com/what-is-tidb/ , which adds SQL, metadata filtering, and distributed scale. This tutorial walks the build path from document loading to a first query, using the LlamaIndex vector store documentation https://developers.llamaindex.ai/python/framework/module guides/storing/vector stores/ and the TiDB integration guide https://docs.pingcap.com/ai/vector-search-integrate-with-llamaindex/ as references. SQL matters here because retrieval rarely stays a pure similarity problem. You filter by tenant, source, or date, and you want those fields in the same system as the vectors. By the end you will have a working TiDBVectorStore, a metadata-filtered query, and a checklist for taking it to production. Key takeaways - The default LlamaIndex vector store lives in memory, so a restart wipes the index unless you persist it yourself. - Swapping in TiDB takes one StorageContext argument. The VectorStoreIndex code you already wrote stays the same. - Metadata filters narrow nearest-neighbor results by fields such as tenant or source, which keeps multi-user retrieval scoped correctly. - TiDBVectorStore supports only LlamaIndex’s default query mode, so keyword and hybrid retrieval need a separate SQL or pytidb path. - TiDB vector search https://docs.pingcap.com/ai/vector-search-overview/ runs on TiDB Cloud Starter https://www.pingcap.com/tidb-cloud-starter/ or TiDB Self-Managed https://www.pingcap.com/tidb/self-managed/ v8.4.0 and later v8.5.0 or later recommended . How to Build a LlamaIndex Vector Store You build a LlamaIndex vector store on TiDB in five steps: load documents, split them into nodes, create a TiDBVectorStore, wrap it in a StorageContext, and build a VectorStoreIndex from it. The index code matches the in-memory version. Only the storage context changes. The table below maps each piece of that flow to what LlamaIndex does by default and what a production setup replaces it with. | Component | Default behavior | Production alternative | TiDB role | |---|---|---|---| | Vector storage | In-memory store inside your Python process | Database-backed vector store | TiDBVectorStore writes embeddings to a TiDB table | | Persistence | Lost on exit unless you persist to local disk | Durable remote table | Data survives restarts and redeploys, and you can inspect it with SQL | | Metadata filters | Applied to in-process data | Applied by the backend during nearest-neighbor search | TiDBVectorStore accepts LlamaIndex MetadataFilters | | Multi-user access | One process owns the index | Shared store reached over the network | Many app instances connect to one cluster with a connection string | | Scale | Bounded by one machine’s RAM | Store that scales out | TiDB distributes rows across TiKV nodes, and vector indexes run on TiFlash replicas | You need three things before you start: Python 3.10 or later, an OpenAI API key for embeddings, and a TiDB cluster. The TiDB docs https://docs.pingcap.com/ai/vector-search-integrate-with-llamaindex/ recommend a TiDB Cloud Starter instance. TiDB Self-Managed also works on v8.4.0 or later, with v8.5.0 or later recommended. Install the packages: pip install llama-index llama-index-vector-stores-tidbvector Load Documents and Create Nodes LlamaIndex reads files into Document objects, then splits them into nodes, which are the chunks that get embedded and retrieved. Attach metadata at this stage, because every node inherits its parent document’s metadata. That is what makes tenant or document-type filters possible later. python import getpass import os from llama index.core import Settings, SimpleDirectoryReader from llama index.core.node parser import SentenceSplitter from llama index.embeddings.openai import OpenAIEmbedding Set the key before you create the embedding model. os.environ "OPENAI API KEY" = getpass.getpass "OpenAI API key: " Pin the embedding model so the vector dimension is known 1536 here . Settings.embed model = OpenAIEmbedding model="text-embedding-3-small" documents = SimpleDirectoryReader "./data/docs" .load data for document in documents: document.metadata "tenant" = "acme" document.metadata "doc type" = "runbook" splitter = SentenceSplitter chunk size=512, chunk overlap=50 nodes = splitter.get nodes from documents documents print f"{len documents } documents became {len nodes } nodes" Connect StorageContext to TiDBVectorStore Create the vector store with your connection string, then hand it to a StorageContext. The vector dimension value must match your embedding model. TiDB vector search accepts vectors of up to 16,383 dimensions https://docs.pingcap.com/ai/vector-search-limitations/ , so common embedding models fit. Copy the PyMySQL connection string from the Connect dialog of your TiDB Cloud Starter instance. python import getpass from llama index.core import StorageContext from llama index.vector stores.tidbvector import TiDBVectorStore Format: mysql+pymysql://<USER :<PASSWORD @<HOST :4000/<DB ?ssl ca=/etc/ssl/cert.pem&ssl verify cert=true&ssl verify identity=true tidb connection string = getpass.getpass "TiDB connection string: " vector store = TiDBVectorStore connection string=tidb connection string, table name="docs vectors", distance strategy="cosine", vector dimension=1536, drop existing table=False, storage context = StorageContext.from defaults vector store=vector store Keep drop existing table=False outside of throwaway experiments. Setting it to True drops the table on startup. Index Data and Run the First Query Pass the nodes and the storage context to VectorStoreIndex. LlamaIndex generates embeddings and writes them to the docs vectors table. From then on, any process with the connection string can reopen the same index without re-embedding anything. python from llama index.core import VectorStoreIndex index = VectorStoreIndex nodes, storage context=storage context, show progress=True query engine = index.as query engine similarity top k=3 response = query engine.query "How do we rotate the API keys?" print response Later, in a new process: reconnect to the stored vectors. index = VectorStoreIndex.from vector store vector store=vector store The TiDB docs example uses VectorStoreIndex.from documents ... , which runs the same load-split-embed-store steps in one call. Splitting nodes yourself, as above, gives you control over chunk size and overlap. For a longer end-to-end example with a local model, see this LlamaIndex RAG implementation guide https://www.pingcap.com/article/building-rag-with-llama3-ollama-llamaindex-tidb-serverless/ . What Does a LlamaIndex Vector Store Actually Store? A LlamaIndex vector store holds one embedding per node, and depending on the backend it can also hold the node text and its metadata. It answers one question: which stored vectors sit closest to this query vector? Everything else, including how you chunk, which filters you apply, and how many results you pass to the LLM, is retrieval policy that you design around it. Documents, Nodes, and Embeddings Four terms cause most of the confusion, so this article uses them the same way throughout. | Term | What it is | Created by | |---|---|---| | Document | A source item as loaded, such as a file or a web page | A reader like SimpleDirectoryReader | | Node | A chunk of a document that carries metadata and a link back to its source | A node parser like SentenceSplitter | | Embedding | The vector that represents one node’s text | Your embedding model | | Metadata | Key-value fields attached to a document or node | You, at load time | The vector store persists embeddings for nodes. It does not persist your original files, so keep those in object storage or wherever they live today. Metadata and Filterable Fields Metadata is how you carry business context into retrieval. Fields such as tenant , source , doc type , or published at let a query say “only nearest neighbors from this customer’s runbooks.” With TiDBVectorStore, the table holds the vector, the node text, and a JSON metadata column, and you can confirm the layout with DESCRIBE docs vectors in any MySQL-compatible client. By default, LlamaIndex also includes metadata in the text it embeds and in the context it sends to the LLM. If a field exists only for filtering, such as an internal tenant ID, exclude it from both by setting excluded embed metadata keys and excluded llm metadata keys on the document. Retrieval Context for RAG and Agents For retrieval-augmented generation, the retrieved nodes become the context in the LLM prompt, so the store’s job is to return the right few chunks. For agents with long-term memory, the same mechanism stores prior conversation turns or extracted facts as nodes, then retrieves them by similarity at the start of a later session. In both cases, storage, indexing, and retrieval policy are separate decisions. A database gives you durable storage and filtering. Chunking, top-k, and filter rules remain your code. When Should You Move From the In-Memory Store to TiDB? Move to a persistent store when the index has to outlive one process, serve more than one user, or hold data you cannot afford to re-embed after every deploy. The default in-memory store is the right tool until one of those conditions shows up. Re-embedding is the hidden cost: every restart means paying your embedding provider again for the same corpus and waiting for it to finish. Prototype Fast with the Default Store For a notebook, a demo, or a corpus that fits comfortably in RAM, the default needs no setup. You can even write it to disk and reload it, which covers a single-developer workflow. python from llama index.core import StorageContext, VectorStoreIndex, load index from storage index = VectorStoreIndex nodes in-memory vector store index.storage context.persist persist dir="./storage" Later, in the same environment: index = load index from storage StorageContext.from defaults persist dir="./storage" Add Persistence for Production Retrieval Local persistence breaks down when the app runs on more than one machine or the corpus keeps growing. Common triggers are: - A corpus too large for one machine’s memory. - Several app replicas that need the same index. - Per-tenant or per-source filtering. - Agents that must remember across sessions. - Deployment pipelines where the container filesystem is disposable. Here is an illustrative example. A support-docs assistant starts as a script over 200 files on a laptop. Six months later it serves 40 customers from three API replicas, each customer sees only its own docs, and new content arrives daily. The in-memory index now needs a shared home, a tenant filter, and incremental inserts. Moving to TiDBVectorStore covers all three without changing the query code, because the replicas call VectorStoreIndex.from vector store ... against one table. Support Long-Term Memory for AI Agents An agent that remembers earlier sessions needs the same properties: durable storage, filtering by user, and retrieval by meaning. A simple pattern writes each memory worth keeping, such as a stated preference or a resolved issue, as a node with user id and kind metadata. At the start of a new session, you query with a filter on that user. For a broader look at what an agent memory database needs, see this guide to choosing a database for AI agents https://www.pingcap.com/compare/best-database-for-ai-agents/ . PingCAP also describes a persistent context pattern for AI agents https://www.pingcap.com/solutions/ai-agent-context/ . python from llama index.core.schema import TextNode memory = TextNode text="User wants all code samples in Python and prefers short answers.", metadata={"user id": "u 123", "kind": "memory", "session id": "s 42"}, index.insert nodes memory What TiDB Adds TiDB combines SQL, vector search, JSON metadata, and horizontal scale in one MySQL-compatible distributed SQL database, which makes it a distributed vector database for RAG https://www.pingcap.com/ai/ without a separate store for each retrieval concern. The TiDB integration overview https://docs.pingcap.com/ai/vector-search-integration-overview/ lists LlamaIndex as an officially supported AI framework, and the LlamaIndex integration guide https://docs.pingcap.com/ai/vector-search-integrate-with-llamaindex/ covers TiDB Cloud Starter and TiDB Self-Managed v8.4.0 or later, with v8.5.0 or later recommended. The same docs mark vector search as a public preview feature, so check the release notes before you plan a launch around it. For more patterns on the same foundation, browse the guide to build AI applications with vector search https://www.pingcap.com/developers/build-ai-apps/ . Keep Operational Data and Vectors Together MySQL compatibility matters here because your application likely already has users, tenants, permissions, and content tables behind MySQL drivers and ORMs. With TiDB, the vector table lives in the same cluster, so you can inspect it with the SQL tools you already use and relate retrieved nodes to business records by key. TiDB shards data across TiKV nodes by key range without application-level sharding, which keeps the same schema working as the corpus grows. Filter and Retrieve in One Backend TiDBVectorStore translates LlamaIndex metadata filters into a SQL WHERE clause over the table’s JSON metadata column, so tenant, source, and document-type constraints are enforced in the database. By default, the filter runs before the distance ordering. According to the TiDB docs, that pattern returns exact results but cannot use a vector index, so cost grows with the number of rows that pass the filter. Metadata filtering is a retrieval-scoping tool, and your application’s authorization layer still decides who may ask for what. TiDB is not the only reasonable backend. If your corpus is small, your team already runs a single PostgreSQL instance, and you have no plans for multi-tenant scale, adding another database may not pay off. TiDB earns its place when you want SQL, vectors, and horizontal scale in one system and your team already works with MySQL tooling. How Do Metadata Filters and Keyword Search Improve LlamaIndex Retrieval? Embeddings rank by meaning, which is the wrong tool when the right answer depends on who owns the data, which document type it came from, or an exact identifier such as an error code. Metadata filters restrict results to rows that match fields such as tenant or document type. Keyword search handles identifiers. Semantic search alone is enough when users ask open-ended questions over a uniform corpus. Filter by Tenant, Source, and Document Type LlamaIndex expresses filters with MetadataFilters . Multiple filters combine with AND by default. python from llama index.core.vector stores.types import MetadataFilter, MetadataFilters filters = MetadataFilters filters= MetadataFilter key="tenant", value="acme", operator="==" , MetadataFilter key="doc type", value="runbook", operator="==" , query engine = index.as query engine filters=filters, similarity top k=3 response = query engine.query "How do we rotate the API keys?" The TiDB docs show how strict this is. With a book = "paul graham" filter on an index where every node has book = "paul graham" , the query returns Empty Response instead of the closest semantic match. An empty result is the correct outcome when nothing passes the filter, so handle it explicitly in your app. Add Keyword Search for Identifiers and Error Codes TiDBVectorStore supports only LlamaIndex’s default query mode, so LlamaIndex-native hybrid mode is not available on this backend. Keyword retrieval has to run as its own path. TiDB’s full-text search provides one: you add a full-text index to a text column and rank matches by BM25 relevance with fts match word . The node text lives in the document column of your vector table. ALTER TABLE docs vectors ADD FULLTEXT INDEX document WITH PARSER MULTILINGUAL ADD COLUMNAR REPLICA ON DEMAND; SELECT id, document FROM docs vectors WHERE fts match word "ERR 4012", document ORDER BY fts match word "ERR 4012", document DESC LIMIT 10; fts match word tokenizes the query and matches any token, so it does not do exact phrase matching. Test how your parser tokenizes identifiers such as ERR 4012 before you rely on it for error-code lookups. Full-text search is still early. It is currently available only on TiDB Cloud Starter in five AWS regions, so check the full-text search docs https://docs.pingcap.com/ai/vector-search-full-text-search-sql/ for the current list before you design around it. The hybrid search guide https://docs.pingcap.com/ai/vector-search-hybrid-search/ shows how the pytidb SDK fuses vector and keyword results with reciprocal rank fusion or weighted scores. To decide whether you need keyword search, run this comparison on your own data: 1. Collect 20 real queries, including some with exact identifiers such as error codes or product SKUs. 2. Run each query three ways: keyword search only, semantic search with metadata filters, and semantic search without filters. 3. Record whether the correct chunk appears in the top five for each method. 4. Add a keyword path only if identifier queries fail under semantic ranking. Reduce Hallucinations With Better Retrieval Context An LLM that receives the wrong chunks will answer from them anyway. Filters remove chunks from the wrong tenant or document type, and keyword matching rescues identifier queries that embeddings blur. Neither guarantees a correct answer, so measure retrieval quality directly instead of judging it from final responses. For questions that depend on relationships between entities, GraphRAG with a knowledge graph https://www.pingcap.com/article/graphrag-enhancing-traditional-rag-through-knowledge-graph/ can add context that chunk similarity misses. Checklist for a Production-Ready Vector Store A production-ready LlamaIndex vector store has deliberate chunking, a pinned embedding model, planned metadata, tuned inserts, measured retrieval, enforced tenant isolation, and logging. Use this list before you ship. - Chunking: Choose chunk size and overlap on purpose. This tutorial uses 512 and 50. Re-test when your document types change. - Embedding model: Pin the model and vector dimension together. Switching models means re-embedding the corpus into a new table. - Metadata design: Decide filterable fields up front, such as tenant, source, doc type, and timestamp. Keep values consistent, and exclude filter-only fields from embeddings. - Insert batch size: Tune it for remote backends and larger loads details below . - Vector index: With tidb-vector 0.0.15, TiDBVectorStore does not create a vector index. For large tables, add an HNSW index yourself, which needs a TiFlash replica. Then run EXPLAIN on a filtered query and look for annIndex: . Filters applied before the distance ordering return exact results but scan every matching row, so cost grows with the filtered set. See the TiDB vector search index https://docs.pingcap.com/ai/vector-search-index/ docs. - Retrieval evaluation: Keep a fixed set of queries with known correct chunks, and track top-k hit rate before and after each change. - Multi-user isolation: Apply the tenant filter on every query in application code. If you need a hard boundary, give each tenant its own table. Partitioned tables cannot have vector indexes, so partitioning by tenant means queries run without one. - Deletes and updates: Plan how removed or changed source documents leave the index. vector store.delete doc id removes a document’s nodes. - Secrets: Read the connection string from a secret manager or environment variable, not a prompt. - Observability: Log each query’s filters, returned node IDs, scores, and latency. The TiDB performance tuning https://docs.pingcap.com/ai/vector-search-improve-performance/ docs cover what to check on large tables. On batch size: the LlamaIndex TiDB example notes that a smaller insert batch size , such as 1000, can avoid exceeding the packet size limit during insertion. Remote databases and large vector loads are where this tends to matter, so lower it if inserts fail with packet-size errors. index = VectorStoreIndex nodes, storage context=storage context, insert batch size=1000, show progress=True, If you are still choosing a backend, this comparison of the best vector databases for LlamaIndex https://www.pingcap.com/compare/best-vector-database/ covers the trade-offs. Sources checked October 30, 2026, with llama-index 0.14.25 and llama-index-vector-stores-tidbvector 0.6.0: the LlamaIndex vector store docs , the LlamaIndex TiDB vector store page , and the TiDB integration guide . Build LlamaIndex Retrieval on TiDB A LlamaIndex vector store on TiDB keeps embeddings, node text, and metadata in one table that survives restarts and that every app replica can query with the same filters. To try it on your own documents, create a free TiDB Cloud Starter instance https://tidbcloud.com/free-trial/ , paste its PyMySQL connection string into the TiDBVectorStore call above, and measure top-k hit rate with the 20-query comparison from the filtering section. If you plan to add keyword search, create the instance in one of the AWS regions where full-text search is available. LlamaIndex Vector Store FAQs What Is a LlamaIndex Vector Store? A LlamaIndex vector store is the storage layer behind a VectorStoreIndex. It keeps embeddings for your nodes and, depending on the backend, the node text and metadata. When you query, LlamaIndex embeds the question, asks the vector store for the nearest nodes, and passes them to the LLM. You choose the backend through a StorageContext, so the same index code works with an in-memory store or a database such as TiDB. Is the Default LlamaIndex Vector Store Persistent? No. By default, LlamaIndex keeps vectors in an in-memory store inside your Python process, so they disappear when it exits. You can write the index to local disk with storage context.persist , which works for single-machine prototypes. Once several services share the index, the corpus outgrows memory, or deployments replace your filesystem, use a database-backed store instead. Can I Use a SQL Database as a LlamaIndex Vector Store? Yes. LlamaIndex supports database-backed vector stores, including TiDB through the llama-index-vector-stores-tidbvector package. TiDB is a MySQL-compatible distributed SQL database with built-in vector search, so one system holds vectors, metadata, and application data. Teams choose a SQL backend for durability, shared access, and filtering with tools they already know. Because TiDBVectorStore supports only LlamaIndex’s default query mode, plan a separate SQL path if you need keyword search. How Does a Vector Store Support Long-Term Memory for AI Agents? Store each memory worth keeping, such as a preference, a decision, or a conversation summary, as a node with metadata like user id and session id . At the start of a new session, embed the user’s message and retrieve the closest memories, filtered to that user. Because the vector store is persistent, memories survive restarts and can be shared across agent instances. Do I Need Metadata Filters for LlamaIndex Retrieval? You need them whenever relevance depends on more than meaning, such as tenant boundaries, document types, sources, or dates. Without filters, a similarity search can return a near match from the wrong customer or an outdated document. Filters improve precision, but they are not a security control on their own, so keep authorization checks in your application. A single-user prototype over a uniform corpus can skip them. Experience modern data infrastructure firsthand. TiDB Cloud Dedicated A fully-managed cloud DBaaS for predictable workloads TiDB Cloud Starter A fully-managed cloud DBaaS for auto-scaling workloads