Chapter 3 questions

Hugging Face tokenizer code in the Chapter 3 course notebook fails because the Datasets library now returns a Column object instead of a plain Python list when indexing by column name, breaking the tokenizer's expected input format. The minimal fix is to wrap dataset column access with `list()` to convert the column to a `list[str]`, as documented in the tokenizer API. The notebook uses the MRPC dataset for sentence-pair classification, where the actual task requires tokenizing both sentences together for BERT.

In terms of code, just list will fix it. Why the notebook code fails now Root cause The tokenizer expects text inputs like: str list str - batched text-pair inputs built from those That is how the tokenizer API is documented today. Hugging Face https://huggingface.co/docs/transformers/main classes/tokenizer But dataset column access is no longer best thought of as “always a plain Python list.” Current Datasets docs explicitly say that indexing by column name first returns a Column object. Hugging Face https://huggingface.co/docs/datasets/access So this line: tokenizer raw datasets "train" "sentence1" can fail because the tokenizer sees a dataset column object , not a plain Python list str . Background: why this is confusing Older Datasets docs described dataset "sentence1" as returning a Python list of values . Hugging Face https://huggingface.co/docs/datasets/v1.11.0/exploring.html Current docs describe column-first indexing as returning a Column object. Hugging Face https://huggingface.co/docs/datasets/access That difference explains why: - older examples could work as written - current environments can reject the same code - the course notebook looks reasonable, but still breaks in practice So the real background is API/documentation drift across library versions and implementations , not that your code idea was conceptually wrong. Hugging Face https://huggingface.co/docs/datasets/v1.11.0/exploring.html Why this notebook uses two sentences at all This dataset is MRPC , a sentence-pair classification task. Each example has: sentence1 sentence2 - a label telling whether they are paraphrases The course explains that for BERT, the correct pair format is: CLS sentence1 SEP sentence2 SEP and that token type ids distinguish the first sentence from the second. GitHub https://raw.githubusercontent.com/huggingface/course/main/chapters/en/chapter3/2.mdx So even when the notebook first tokenizes sentence1 and sentence2 separately, that is only an introductory step. The actual task is to tokenize the pair together. GitHub https://raw.githubusercontent.com/huggingface/course/main/chapters/en/chapter3/2.mdx Minimal fix for the notebook The smallest code change is to convert each dataset column to a plain list. Replace this tokenized sentences 1 = tokenizer raw datasets "train" "sentence1" tokenized sentences 2 = tokenizer raw datasets "train" "sentence2" With this tokenized sentences 1 = tokenizer list raw datasets "train" "sentence1" tokenized sentences 2 = tokenizer list raw datasets "train" "sentence2" And replace this: tokenized dataset = tokenizer raw datasets "train" "sentence1" , raw datasets "train" "sentence2" , padding=True, truncation=True, with this: tokenized dataset = tokenizer list raw datasets "train" "sentence1" , list raw datasets "train" "sentence2" , padding=True, truncation=True, Why this works list ... converts the dataset column into the kind of container the tokenizer is documented to accept: a plain list str . Hugging Face https://huggingface.co/docs/transformers/main classes/tokenizer