Hmm… maybe something like this? :
I think I would separate the three questions a little, because they interact, but they do not have to share the same answer.
My short version would be:
There is also one design choice that affects all three: whether the multilingual version is meant to be a faithful translation of the English v1 interpretation, a locale-specific interpretation that is allowed to adapt, or both.
Those are slightly different datasets, even if they can live in the same repository.
For the Hub side, language-specific configs seem perfectly reasonable. Hugging Face describes a subset/configuration as a sub-dataset, and explicitly gives multilingual datasets with one subset per language as a typical use case. The repository can define those configs directly in the dataset card YAML and map them to different files or globs without custom code; the current data-files configuration docs show that pattern.
The useful distinction, I think, is:
How the multilingual data is canonically aligned does not have to be identical to how users load it from the Hub.
A nice precedent is BOUQuET. Its underlying data is fully multi-way parallel, but it is distributed through several views/configs, including language-specific ones. So using configs does not mean giving up multi-way alignment.
If I were trying a first v2 shape, I would probably test something like this:
data/
en.parquet
tr.parquet
es.parquet
de.parquet
pt-BR.parquet
with a consistent schema in every file, and then expose en
, tr
, es
, de
, pt-BR
, plus optionally an all
config.
Conceptually:
card_number
card_id? # optional; card_number already works as an ID
locale
card_name
upright_meaning
reversed_meaning
love_meaning
career_meaning
variant_type? # e.g. faithful_translation / localized_interpretation
creation_method? # human / machine / machine_postedited
review_status?
source_revision?
I would not call that the only correct representation, though.
There are at least three sensible shapes:
| Priority | Shape I would test first |
|---|---|
| Load one language easily; add languages later | one common schema per locale + Hub configs |
| Inspect all five translations side by side | wide/parallel language columns |
| Treat each card as one self-contained object | nested translations structure |
| Filter/analyze provenance per locale | card × locale long/tidy representation |
At only 78 cards, one unusually cheap way to decide is to prototype maybe five cards in two or three shapes, put them through the Dataset Viewer/Data Studio, and freeze the schema only after seeing which one is easiest to inspect and query.
That is probably more informative than deciding from schema aesthetics alone.
Also, I would keep the storage format as a separate decision. CSV is fine while most values are scalar strings. If v2 starts using real arrays for keywords or nested provenance/translation objects, Parquet or JSONL may become less awkward. Hugging Face currently supports all of these and generally recommends Parquet when richer typed/tabular data is useful.
I would keep the compact meanings.
They are not merely an incomplete version of a future paragraph. They are a useful representation in their own right: they are easy to compare, embed, retrieve, classify, audit manually, and translate at matching granularity.
If more context becomes useful, I would add it rather than replace the compact layer.
Something like:
upright_keywords
upright_meaning
upright_context
where these could mean:
keywords:
["new beginnings", "innocence", "adventure", "free spirit"]
compact meaning:
"New beginnings, innocence, adventure, free spirit"
context:
a longer explanatory paragraph
Not every layer necessarily has to exist in v2. The point is just that they answer different experimental questions.
A very close independent example is the Tarotoo 78-card dataset, which separates keywords_upright
/ keywords_reversed
arrays from prose meaning fields and contextual fields. I would use that only as a useful schema comparison, not as an authority on which interpretation of a card is “correct.”
There is also a broader NLP reason to preserve compact descriptions: natural-language label descriptions and related terms are useful inputs for zero-shot and description-based classification rather than just human-readable metadata. For example, Gao et al. (EMNLP 2023) study training with natural-language label descriptions, including related terms and short descriptions.
One thing I would try to preserve quite carefully is granularity alignment.
For example, comparing:
English: 4 compact concepts
Spanish: a 120-word explanatory paragraph
with an embedding model makes it difficult to know whether a difference comes from translation drift or simply from comparing different kinds of text.
For experiments, I would prefer:
keywords ↔ keywords
compact meaning ↔ compact meaning
context paragraph ↔ context paragraph
whenever possible.
This is the part where I would be most cautious about reducing everything to one score.
I do not know of a standard metric specifically for “tarot symbolic consistency,” but the problem can be decomposed into several ordinary, testable questions.
A small stack could be:
structural alignment
↓
same-card cross-lingual retrieval
↓
deliberate corruption / contrast tests
↓
selective bilingual human review
↓
optional deck-level relational analysis
This part is nearly free and catches boring errors before any semantic metric is involved.
For example, for five locales:
78 cards per locale
unique(card_number, locale)
no missing compact meanings
expected locale set
valid categorical values
static metadata consistent where it should be
If the dataset becomes 78 × 5 aligned rows, simple build-time checks may prevent a surprising amount of later debugging.
For each field, treat the 78 meanings in one language as a tiny retrieval corpus.
For example:
English Fool / upright
↓
search all 78 Spanish upright meanings
↓
where does Spanish Fool rank?
Then repeat for all 78 cards and in both directions.
Useful summaries could be:
This is not a particularly exotic evaluation. Sentence Transformers has a TranslationEvaluator built around essentially this idea: aligned source item
i
and target item i
should retrieve one another among all candidates.I would interpret it as an alignment sanity check, not as proof of translation quality.
This may be more useful than choosing the “best” embedding metric immediately.
Before trusting a metric, I would ask whether it responds correctly when the data is intentionally damaged.
For example:
easy:
random wrong card
completely wrong target language
medium:
upright ↔ reversed
love ↔ career
remove an important concept
hard:
substitute a semantically similar card
substitute a card from the same suit
preserve most concepts but reverse one important one
Then a very basic desired property is:
score(correct pair) > score(controlled corruption)
or, for retrieval, that the correct card ranks above the corrupted/hard-negative candidate.
This is similar in spirit to DEMETR, which evaluates machine-translation metrics by introducing controlled semantic, syntactic, and morphological perturbations and testing whether the metric actually notices them. One useful lesson from that work is that different learned metrics are sensitive to different error types.
So for this dataset I would probably think of the controls as tests of the evaluator, not just tests of the translations.
Random wrong-card negatives may actually be too easy. A more informative negative is often a card whose English meaning is already nearby.
Because there are only 78 cards, human checking is unusually feasible, but I would still not start by designing a large annotation project.
A lower-cost flow could be:
automatic alignment/retrieval checks
↓
flag:
very bad rank
very small or negative hard-negative margin
disagreement between embedding models
off-target / mixed-language text
↓
bilingual review of flagged items
+ a small random sample of unflagged items
For human review, I would separate at least two questions:
And, if localization is intentional:
That distinction is close to the motivation behind XSTS, a bilingual human-evaluation protocol that focuses explicitly on semantic correspondence/adequacy rather than folding everything into fluency.
Since 78 cards is tiny, another interesting exploratory test is to build the full 78×78 semantic-similarity matrix in each language.
There are only 3,003 unique card pairs, so one can ask whether relative neighborhoods are approximately preserved:
English deck similarity structure
vs
Spanish deck similarity structure
This could reveal broad drift that same-card retrieval misses.
I would keep this explicitly exploratory, though. The result depends heavily on the embedding model, so I would not call a matrix correlation a “symbolic fidelity score.”
[/details]
I think this decision should probably come before finalizing the evaluation.
If the goal is:
Then the English v1 meaning is the reference interpretation, and divergence is mostly something to inspect.
English v1
→ faithful ES
→ faithful DE
→ faithful TR
→ faithful PT-BR
Same-card retrieval and semantic adequacy make sense as fairly direct checks.
Then some divergence from English may be correct behavior.
A Spanish, Turkish, German, or Brazilian Portuguese interpretation may choose a different idiom, emphasis, or culturally natural wording while still serving the local reader better.
In that case, maximizing similarity to English can accidentally punish good localization.
This may actually be the most research-friendly version if it is not too much work:
faithful_translation
localized_interpretation
as separate variants.
Then the dataset can support two different experiments:
A useful precedent here is MASSIVE, which explicitly distinguishes whether an item was a translation
, a localization
, or left unchanged
, rather than treating all target-language differences as one operation.
I would not copy MASSIVE’s annotation machinery, but that distinction seems valuable here.
And I would probably split the metadata into two separate questions:
variant_type:
faithful_translation
localized_interpretation
creation_method:
human
machine
machine_postedited
review_status:
...
because “what relationship does this target text have to the English meaning?” and “how was this text produced?” are not the same thing.
A small synthetic sanity check I triedSo, if I were choosing the lowest-cost path, I think I would do roughly this:
1. Keep the existing 78-card identity and compact meanings.
2. Decide whether v2 represents:
faithful translations,
localized interpretations,
or two separate variants.
3. Prototype a few cards in 2–3 multilingual schemas.
4. Pick the representation that is easiest to inspect,
but expose language-specific Hub configs regardless if useful.
5. Add longer context only as an additional layer.
6. Record just enough provenance to distinguish
locale / variant / creation method / review state.
7. Add tiny structural validation.
8. Evaluate with:
same-card retrieval
+ deliberate hard controls
+ selective human review.
9. Treat deck-level geometry as exploratory, not as the definition
of symbolic consistency.
I do not think this needs to become a large ontology or a heavyweight MT benchmark to be useful.
The small, closed 78-card inventory is actually one of the attractive parts of the dataset: every item can be aligned, audited, perturbed, and compared exhaustively.
So I would mostly preserve that simplicity, and just separate the boundaries that become important once multiple languages enter the picture:
card identity
≠ localized wording
canonical alignment
≠ Hub presentation
compact descriptor
≠ contextual prose
faithful translation
≠ localization
automatic anomaly score
≠ human judgment
That seems enough to keep the current dataset easy to use while making the multilingual version much easier to evaluate and extend later.