Note: This tutorial picks up where Chroma's official quick start guide leaves off.
So you've finished Chroma's quick start — congratulations. Everything works, right up until you feed Chroma some Chinese text.
Then it doesn't.
Chroma's default embedding model, all-MiniLM-L6-v2, is English-only. Given Chinese or Japanese input, it embeds it as though the language were noise. You need a model that was trained on the language you care about.
You have two ways to get one:
| Local model | Online model | |
|---|---|---|
| Runs where | on your machine | vendor's server |
| Network | not needed | required |
| Setup | install packages, download weights | an API key |
| Cost | disk space (hundreds of MB) | per query |
| Offline | works | does not |
This tutorial walks through both, and then a third option that fixes a problem neither of them solves: ranking.
To be concrete, we'll assume Chinese is the language you need to add. Swap the model name and the same code supports any language.
Whatever you choose, the hook into Chroma is the same: the optional embedding_function argument of client.create_collection().
ef = YourEmbeddingFunction(...)
collection = client.create_collection(
name="my_collection",
embedding_function=ef)
That's the whole interface. The rest of this tutorial is about what to put in place of YourEmbeddingFunction.
Assuming you want Chinese support from a local model, these are the usual candidates:
| Model | Language | Dim. | Notes |
|---|---|---|---|
paraphrase-multilingual-MiniLM-L12-v2 |
Multilingual | 384 | lightweight |
BAAI/bge-small-zh-v1.5 |
Chinese | 512 | good for Chinese |
BAAI/bge-base-zh-v1.5 |
Chinese | 768 | same family, stronger |
BAAI/bge-m3 |
Multilingual | 1024 | multilingual |
text2vec-base-chinese |
Chinese | 768 | |
m3e-base |
Chinese | 768 | |
all-MiniLM-L6-v2 |
English | 384 | Chroma's default (for reference) |
Roughly:
More dimensions means better quality and a bigger download.
bge-m3 is 1024-dimensional and will cost you well over 2 GB.bge-small-zh-v1.5 is the cheapest thing that works properly, so that's what we'll use below.
To use a local model you need one extra Python package, sentence-transformers. Chroma deliberately does not ship it, for a good reason:
embedding_functions.SentenceTransformerEmbeddingFunction. sentence-transformers package.torch (PyTorch)transformers`` huggingface-hub``tokenizers`` numpy, scipy and friends.
That's a deep-learning stack — hundreds of MB of dependencies. If your environment already has PyTorch, fine. If your project is, say, a small web service that just happens to need a vector store, that stack is a bit of overkill. So Chroma makes you opt in:
pip install sentence-transformers
Now we can actually build the embedding function.
The first time you construct SentenceTransformerEmbeddingFunction('BAAI/bge-small-zh-v1.5'), it downloads the weights from huggingface.co, where the model lives by default.
If you live in China, or anywhere the connection to huggingface.co is poor, point it at a mirror first. On Linux:
export HF_ENDPOINT="https://hf-mirror.com"
On Windows PowerShell:
$env:HF_ENDPOINT="https://hf-mirror.com"
Or set it from inside Python, which is often the most convenient:
from chromadb.utils import embedding_functions
import os
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
model_name="BAAI/bge-small-zh-v1.5")
The output is like this:
D:\app\anaconda3\envs\ai\lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
weights: 100%|███████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 9559.74it/s]
What to notice in that output:
By default the model lands in ~/.cache/huggingface/hub on Linux, or C:\Users\<you>\.cache\huggingface\hub on Windows.
A user-level cache is convenient, but it makes the project non-portable: a teammate clones the repo, and still has to download 100 MB before anything runs. Down into the project directory instead fixes that — one hf command does it:
$env:HF_ENDPOINT="https://hf-mirror.com"
pip install "huggingface_hub[cli]"
hf download BAAI/bge-small-zh-v1.5 --local-dir ./models/bge-small-zh-v1.5
Then point model_name at the local path. The leading ./ is what tells SentenceTransformerEmbeddingFunction this is a directory and not a HuggingFace model ID:
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
model_name="./models/bge-small-zh-v1.5")
The output:
weights: 100%|██████████████████████████████████████████████████████████████| 71/71 [00:00<00:00, 14191.55it/s]
You can tell it worked: the is faster (no network round-trip to resolve the model), and ./models/bge-small-zh-v1.5 now contains roughly 184 MB of files.
Curious about the internal workings? Let's take a look at the source code:
import inspect
print(inspect.getsource(embedding_functions.SentenceTransformerEmbeddingFunction))
The output is long, so only the beginning matters here:
class SentenceTransformerEmbeddingFunction(EmbeddingFunction[Documents]):
models: Dict[str, Any] = {}
def __init__(
self,
model_name: str = "all-MiniLM-L6-v2",
device: str = "cpu",
normalize_embeddings: bool = False,
**kwargs: Any,
):
...
Two things worth noticing.
model_name is Chroma's English default — which is the whole reason this tutorial exists.device argument: pass "cuda" to move embedding onto a GPU if you have one.
Time to put it to work. If you've run this before, the collection already exists and Chroma will refuse to create it again — hence the commented-out cleanup line, which you'll want in a notebook where cells get re-run:
#client.delete_collection(name='demo1')
python
import chromadb
client = chromadb.Client()
collection = client.create_collection(
name="demo1",
embedding_function=ef)
Now add some documents and query them. These are short, self-contained sentences about programming languages and machine learning — deliberately, so that "what is Transformer?" has a defensible right answer:
docs = [
'Python 是一种流行的编程语言，广泛用于数据处理、机器学习以及广义上的人工智能。',
'数据处理包括将原始数据清洗、转换和聚合成有用的格式。',
'机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。',
'深度学习使用具有多层的神经网络来学习分层表示。',
'神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。',
'Transformer 使用自注意力来建模序列中词元之间的关系。',
'注意力机制让模型在生成每个输出表示时，对输入的不同部分进行加权。',
'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
'大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。',
"预训练在针对特定任务进行微调之前，从大规模无标注数据中学习通用表示。",
"微调使用有标注数据或指令数据，使预训练模型适应特定任务或领域。",
"后训练通过指令微调、偏好优化或安全对齐，使预训练模型适应特定需求。",
"检索增强生成（RAG）将来自外部知识源的检索与大语言模型（LLM）的生成相结合。",
"一个基础的 RAG 流程会切分文档、对文本块进行嵌入、将其存入向量索引、检索前 k 个文本块，并将它们传给 LLM。",
"向量嵌入将文本、图像或其他数据映射到高维空间中的点，以进行相似度搜索。",
]
Here, each item of docs represents a document, albeit a short one.
Add documents to the collection:
collection.add(ids=[f'id{i+1}' for i in range(len(docs))], documents=docs)
And ask a question:
resp = collection.query(query_texts=['什么是Transformer？'], n_results=3)
The return value resp looks like this:
{'ids': [['id8', 'id6', 'id9']],
'embeddings': None,
'documents': [['GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
'Transformer 使用自注意力来建模序列中词元之间的关系。',
'大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。']],
'uris': None,
'included': ['metadatas', 'documents', 'distances'],
'data': None,
'metadatas': [[None, None, None]],
'distances': [[0.3059161305427551, 0.36248522996902466, 0.4628119468688965]]}
id8 and id6, are both genuinely about Transformers. Good. id9, is about LLMs, which was not asked about at all.
The distances also tell a story: 0.31 for the top hit is a fairly loose match. The model is unsure, and the ranking quietly disagrees with any human reading of the question.
The cause is worth naming, because it recurs. An embedding model compresses a whole sentence into a single vector, and the vector is dominated by what words appear, not by what the sentence is doing with them. id8 is stuffed with the token "Transformer"; id6 uses it once. For a similarity search over a bag of tokens, id8 simply looks more relevant. Nothing in the embedding stage distinguishes "mentions Transformer" from "explains Transformer" — that distinction requires reading the query and the candidate together, which is a different architecture. We'll come back to it in the last section.
If you'd rather not manage packages and downloads, a hosted embedding API is the alternative. The trade-offs are the ones from the table at the top: you need an API key, and you pay per call, but there's nothing to install and the model is somebody else's problem to keep up to date.
Chroma doesn't know about your vendor, so you write a small adapter. All it needs is a class with a __call__ method that maps a list of strings to a list of vectors.
Here we use Alibaba's DashScope, whose API is OpenAI-compatible — which means the official openai client works as-is, with only the base_url changed:
import os
from openai import OpenAI
from chromadb.api.types import EmbeddingFunction, Embeddings
class DashScopeEmbeddingFunction(EmbeddingFunction):
def __init__(self, api_key : str = None, model : str = "qwen3.7-text-embedding-flash"):
self.model = model
self.client = OpenAI(
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
api_key=api_key or os.getenv("DASHSCOPE_API_KEY"))
def __call__(self, input: list[str]) -> Embeddings:
response = self.client.embeddings.create(
model=self.model,
input=input,
)
return [item.embedding for item in response.data]
A few notes on the adapter:
DASHSCOPE_API_KEY environment variable when you don't pass one in. Never hard-code it in a notebook you might share.__call__ receives a .embedding and .index; if you ever need to be defensive about ordering, sort by index before returning.
Then build a second collection on top of it:
#Run this if it complains about the collection 'demo2' already exists
#client.delete_collection(name='demo2')
python
import chromadb
client = chromadb.Client()
ds_ef = DashScopeEmbeddingFunction()
collection2 = client.create_collection(
name="demo2",
embedding_function=ds_ef)
The same documents, the same query — only the model changed:
collection2.add(ids=[f'id{i+1}' for i in range(len(docs))],
documents=docs)
collection2.query(query_texts=['什么是Transformer？'], n_results=3)
{'ids': [['id6', 'id8', 'id3']],
'embeddings': None,
'documents': [['Transformer 使用自注意力来建模序列中词元之间的关系。',
'GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。',
'机器学习通过训练模型来发现数据中的模式，并对新样本进行预测。']],
'uris': None,
'included': ['metadatas', 'documents', 'distances'],
'data': None,
'metadatas': [[None, None, None]],
'distances': [[0.8229950666427612, 0.8484762907028198, 1.2087467908859253]]}
Better. id6 — the sentence that actually defines the Transformer — now comes first, and id8 second. The ordering finally matches what a human would say.
Don't read too much into the third hit, id3 (about machine learning): it's an artefact of asking for three results when only two documents are genuinely on topic. The third slot has to be filled by something, and something marginally related is the best available answer. Nevertheless, its distance ~1.21 is much larger than the first two ~0.82 and ~0.85.
One caution on the numbers: distances are not comparable across models. The local model returned ~0.31 and this one returns ~0.82 for its best hit, but that does not mean it is worse. Every model has its own scale and its own notion of distance; compare rankings, never raw scores, when models differ.
So we've fixed the language problem, and we now have a ranking that looks sensible. But it took trying two models to get there, and nothing guarantees the third query works as well.
The underlying weakness hasn't gone away. The embedding stage still compares the query against each document separately, through a lossy single vector. That's what makes it fast — the whole corpus can be indexed once, offline — and it's also what makes it approximate.
A reranker replaces that approximation with a slow, careful look:
Because the reranker sees query and document jointly, it can tell "mentions Transformer" from "explains Transformer" — exactly the distinction the embedding model blurred. The price is that it can't be precomputed: it must run on every query against every candidate, which is why it's used to reorder a shortlist rather than to search the corpus.
Rerankers are also much bigger. BAAI/bge-reranker-v2-m3 wants about 2.3 GB of disk, which is a lot to carry for a ranking refinement. So here we'll use a hosted one, qwen3.7-text-rerank.
The embedding model and the reranker are completely independent — nothing requires them to come from the same vendor, or even to be the same kind of thing. Here, the retrieval comes from the local bge-small-zh-v1.5 and the reranking from the online qwen3.7-text-rerank.
We keep using collection from Option 1, and this time ask for 10 candidates instead of 3:
query = '什么是Transformer？'
res = collection.query(
query_texts=[query],
n_results=10)
candidates = res['documents'][0]
ids = res['ids'][0]
for s in candidates[:3]: print(s)
GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transformer，是一种仅使用解码器的 Transformer 模型，经过训练以预测下一个词元。
Transformer 使用自注意力来建模序列中词元之间的关系。
大型语言模型（LLM）是在海量文本语料库上训练的神经网络，用于预测下一个词元。
Note which sentence is on top. ids is carried along in parallel with candidates, because the reranker returns positions into the list you handed it — you'll need those IDs to report results afterward.
The DashScope API is not OpenAI-shaped for reranking, so this step uses its own SDK:
#!pip install dashscope
python
import dashscope
dashscope_workspace_id = 'ws-xxx' # your own workspace ID
dashscope.base_http_api_url = f'https://{dashscope_workspace_id}.cn-beijing.maas.aliyuncs.com/api/v1'
rerank_resp = dashscope.TextReRank.call(
model="qwen3.7-text-rerank",
query=query,
documents=candidates,
top_n=3)
Note the shape of the call: one query, many documents, and a top_n that decides how many survive. query and candidates are reusing the variables from the previous section.
for item in rerank_resp.output.results:
print(f"[{item.relevance_score:.3f}] {ids[item.index]}. {candidates[item.index][:60]}...")
[0.851] id6. Transformer 使用自注意力来建模序列中词元之间的关系。...
[0.780] id8. GPT 全称为 Generative Pre-trained Transformer，即生成式预训练 Transform...
[0.509] id5. 神经网络是受大脑启发的计算系统，由相互连接的节点层组成，这些节点层学习从数据中识别模式并进行预测。...
There it is. The definition sentence for "Transformer" is now first, the GPT sentence second — a gap of 0.851 vs 0.780, where the embedding model had them within 0.03 of each other and in the wrong order.
Two details in that code are worth keeping in mind:
item.index is a position in candidates, so candidates[item.index] recovers the text and ids[item.index] recovers the Chroma ID. That parallel-list lookup is the only fiddly part of using a reranker.distances from either embedding model. They're the reranker's own relevance scale — only useful for ordering, and only within a single call.
The recipes above are the pieces; in practice they compose into one retrieval rule.
def retrieve(query, k=3):
res = collection.query(query_texts=[query], n_results=max(k * 4, 10))
candidates, ids = res['documents'][0], res['ids'][0]
resp = dashscope.TextReRank.call(
model="qwen3.7-text-rerank",
query=query,
documents=candidates, top_n=k)
return [(ids[r.index], candidates[r.index], r.relevance_score)
for r in resp.output.results]
Two heuristics are hidden in those numbers. The k * 4 (with a floor of 10) is the retrieval depth: fetch several times what you need, so that the reranker has something worth reordering, but not so much that you're paying for the whole corpus twice. And when there are fewer candidates than k, the reranker simply returns what it got — no special case needed.
The three options, and when each makes sense:
| Best for | Cost | Watch out for | |
|---|---|---|---|
| Local model | privacy, offline use, tight budgets | ~184 MB of disk | needs a deep-learning stack installed |
| Online model | fastest to a working demo, best quality per line of code | per-query fees, API key | vendor lock-in, network dependency |
| Reranker | any retrieval where ranking matters | a second model in the loop | latency — it runs on every query |
The thread running through all of it: embeddings get you a shortlist, they don't get you an answer.
An embedding model has to compress a sentence into one vector, and it must do so before it ever sees your query — which is precisely why it's fast, and precisely why it confuses "mentions the topic" with "answers the question." A reranker fixes that by reading query and candidate together, at the cost of doing the work fresh each time. Retrieve broad, rerank narrow, then generate.
Swap BAAI/bge-small-zh-v1.5 for any other row in the model table and the code above is unchanged — including the all-MiniLM-L6-v2 you started with, back in its natural habitat of English text.