{"slug": "chapter-3-questions", "title": "Chapter 3 questions", "summary": "Hugging Face tokenizer code in the Chapter 3 course notebook fails because the Datasets library now returns a Column object instead of a plain Python list when indexing by column name, breaking the tokenizer's expected input format. The minimal fix is to wrap dataset column access with `list()` to convert the column to a `list[str]`, as documented in the tokenizer API. The notebook uses the MRPC dataset for sentence-pair classification, where the actual task requires tokenizing both sentences together for BERT.", "body_md": "In terms of code, just `list()`\n\nwill fix it.\n\nWhy the notebook code fails now\n\nRoot cause\n\nThe tokenizer expects text inputs like:\n\n`str`\n\n`list[str]`\n\n- batched text-pair inputs built from those\n\nThat is how the tokenizer API is documented today. ([Hugging Face](https://huggingface.co/docs/transformers/main_classes/tokenizer))\n\nBut dataset column access is no longer best thought of as “always a plain Python list.” Current Datasets docs explicitly say that **indexing by column name first returns a **`Column`\n\nobject. ([Hugging Face](https://huggingface.co/docs/datasets/access))\n\nSo this line:\n\n```\ntokenizer(raw_datasets[\"train\"][\"sentence1\"])\n```\n\ncan fail because the tokenizer sees a **dataset column object**, not a plain Python `list[str]`\n\n.\n\nBackground: why this is confusing\n\nOlder Datasets docs described `dataset[\"sentence1\"]`\n\nas returning a **Python list of values**. ([Hugging Face](https://huggingface.co/docs/datasets/v1.11.0/exploring.html))\n\nCurrent docs describe column-first indexing as returning a `Column`\n\nobject. ([Hugging Face](https://huggingface.co/docs/datasets/access))\n\nThat difference explains why:\n\n- older examples could work as written\n- current environments can reject the same code\n- the course notebook looks reasonable, but still breaks in practice\n\nSo the real background is **API/documentation drift across library versions and implementations**, not that your code idea was conceptually wrong. ([Hugging Face](https://huggingface.co/docs/datasets/v1.11.0/exploring.html))\n\nWhy this notebook uses two sentences at all\n\nThis dataset is **MRPC**, a sentence-pair classification task. Each example has:\n\n`sentence1`\n\n`sentence2`\n\n- a label telling whether they are paraphrases\n\nThe course explains that for BERT, the correct pair format is:\n\n```\n[CLS] sentence1 [SEP] sentence2 [SEP]\n```\n\nand that `token_type_ids`\n\ndistinguish the first sentence from the second. ([GitHub](https://raw.githubusercontent.com/huggingface/course/main/chapters/en/chapter3/2.mdx))\n\nSo even when the notebook first tokenizes `sentence1`\n\nand `sentence2`\n\nseparately, that is only an introductory step. The **actual task** is to tokenize the pair together. ([GitHub](https://raw.githubusercontent.com/huggingface/course/main/chapters/en/chapter3/2.mdx))\n\nMinimal fix for the notebook\n\nThe smallest code change is to convert each dataset column to a plain list.\n\nReplace this\n\n```\ntokenized_sentences_1 = tokenizer(raw_datasets[\"train\"][\"sentence1\"])\ntokenized_sentences_2 = tokenizer(raw_datasets[\"train\"][\"sentence2\"])\n```\n\nWith this\n\n```\ntokenized_sentences_1 = tokenizer(list(raw_datasets[\"train\"][\"sentence1\"]))\ntokenized_sentences_2 = tokenizer(list(raw_datasets[\"train\"][\"sentence2\"]))\n```\n\nAnd replace this:\n\n```\ntokenized_dataset = tokenizer(\n    raw_datasets[\"train\"][\"sentence1\"],\n    raw_datasets[\"train\"][\"sentence2\"],\n    padding=True,\n    truncation=True,\n)\n```\n\nwith this:\n\n```\ntokenized_dataset = tokenizer(\n    list(raw_datasets[\"train\"][\"sentence1\"]),\n    list(raw_datasets[\"train\"][\"sentence2\"]),\n    padding=True,\n    truncation=True,\n)\n```\n\nWhy this works\n\n`list(...)`\n\nconverts the dataset column into the kind of container the tokenizer is documented to accept: a plain `list[str]`\n\n. ([Hugging Face](https://huggingface.co/docs/transformers/main_classes/tokenizer))", "url": "https://wpnews.pro/news/chapter-3-questions", "canonical_source": "https://discuss.huggingface.co/t/chapter-3-questions/6800?page=9#post_173", "published_at": "2026-07-27 09:07:15+00:00", "updated_at": "2026-07-27 09:07:53.369872+00:00", "lang": "en", "topics": ["developer-tools", "natural-language-processing"], "entities": ["Hugging Face", "Datasets", "MRPC", "BERT"], "alternates": {"html": "https://wpnews.pro/news/chapter-3-questions", "markdown": "https://wpnews.pro/news/chapter-3-questions.md", "text": "https://wpnews.pro/news/chapter-3-questions.txt", "jsonld": "https://wpnews.pro/news/chapter-3-questions.jsonld"}}