# Chapter 3 questions

> Source: <https://discuss.huggingface.co/t/chapter-3-questions/6800?page=9#post_173>
> Published: 2026-07-27 09:07:15+00:00

In terms of code, just `list()`

will fix it.

Why the notebook code fails now

Root cause

The tokenizer expects text inputs like:

`str`

`list[str]`

- batched text-pair inputs built from those

That is how the tokenizer API is documented today. ([Hugging Face](https://huggingface.co/docs/transformers/main_classes/tokenizer))

But dataset column access is no longer best thought of as “always a plain Python list.” Current Datasets docs explicitly say that **indexing by column name first returns a **`Column`

object. ([Hugging Face](https://huggingface.co/docs/datasets/access))

So this line:

```
tokenizer(raw_datasets["train"]["sentence1"])
```

can fail because the tokenizer sees a **dataset column object**, not a plain Python `list[str]`

.

Background: why this is confusing

Older Datasets docs described `dataset["sentence1"]`

as returning a **Python list of values**. ([Hugging Face](https://huggingface.co/docs/datasets/v1.11.0/exploring.html))

Current docs describe column-first indexing as returning a `Column`

object. ([Hugging Face](https://huggingface.co/docs/datasets/access))

That difference explains why:

- older examples could work as written
- current environments can reject the same code
- the course notebook looks reasonable, but still breaks in practice

So the real background is **API/documentation drift across library versions and implementations**, not that your code idea was conceptually wrong. ([Hugging Face](https://huggingface.co/docs/datasets/v1.11.0/exploring.html))

Why this notebook uses two sentences at all

This dataset is **MRPC**, a sentence-pair classification task. Each example has:

`sentence1`

`sentence2`

- a label telling whether they are paraphrases

The course explains that for BERT, the correct pair format is:

```
[CLS] sentence1 [SEP] sentence2 [SEP]
```

and that `token_type_ids`

distinguish the first sentence from the second. ([GitHub](https://raw.githubusercontent.com/huggingface/course/main/chapters/en/chapter3/2.mdx))

So even when the notebook first tokenizes `sentence1`

and `sentence2`

separately, that is only an introductory step. The **actual task** is to tokenize the pair together. ([GitHub](https://raw.githubusercontent.com/huggingface/course/main/chapters/en/chapter3/2.mdx))

Minimal fix for the notebook

The smallest code change is to convert each dataset column to a plain list.

Replace this

```
tokenized_sentences_1 = tokenizer(raw_datasets["train"]["sentence1"])
tokenized_sentences_2 = tokenizer(raw_datasets["train"]["sentence2"])
```

With this

```
tokenized_sentences_1 = tokenizer(list(raw_datasets["train"]["sentence1"]))
tokenized_sentences_2 = tokenizer(list(raw_datasets["train"]["sentence2"]))
```

And replace this:

```
tokenized_dataset = tokenizer(
    raw_datasets["train"]["sentence1"],
    raw_datasets["train"]["sentence2"],
    padding=True,
    truncation=True,
)
```

with this:

```
tokenized_dataset = tokenizer(
    list(raw_datasets["train"]["sentence1"]),
    list(raw_datasets["train"]["sentence2"]),
    padding=True,
    truncation=True,
)
```

Why this works

`list(...)`

converts the dataset column into the kind of container the tokenizer is documented to accept: a plain `list[str]`

. ([Hugging Face](https://huggingface.co/docs/transformers/main_classes/tokenizer))