# Why Random Forest crushed Linear Regression for my IMDb score

> Source: <https://promptcube3.com/en/threads/6075/>
> Published: 2026-08-12 20:46:19+00:00

# Why Random Forest crushed Linear Regression for my IMDb score

## The Setup and the Crash

I built this as a real-world exercise in prompt engineering for data cleaning and basic ML deployment. My goal was to see if I could predict a film's score based on metadata. I used a standard scikit-learn pipeline, but I hit a wall during the preprocessing stage.

The first issue was a classic `ValueError`

when I tried to fit the Linear Regression model. I hadn't handled the categorical variables (like Genre) correctly, and the model choked on the strings.

```
ValueError: could not convert string to float: 'Action'
```

I fixed this using one-hot encoding, but then I ran into a performance bug. My dataset had a few extreme outliers—movies with massive budgets but 1-star ratings—and the Linear Regression model was being pulled wildly off course by them.

## Comparing the Results

Since I wanted a deep dive into why one worked better than the other, I tracked a few specific metrics. I can't use a table here, so here is the breakdown of how they performed on the test set:

**Linear Regression MAE:** 1.42 (way too high for a 1-10 scale)**Random Forest MAE:** 0.68 (much closer to the actual scores)**Linear Regression R² Score:** 0.31**Random Forest R² Score:** 0.74

The Random Forest model won because it handles non-linear relationships and interactions between features much better. For example, the interaction between "Director Reputation" and "Genre" is complex; a horror movie might be rated highly for being "scary," whereas a drama is rated for "acting." Linear Regression tries to find a global average, whereas the decision trees in Random Forest can isolate these specific pockets of data.

## My Practical Tutorial for Implementation

If you're trying to replicate this or doing a similar LLM agent project for data analysis, here is the basic logic I used for the Random Forest implementation:

1. Load the IMDb dataset and drop rows with missing values.

2. Encode categorical features using `pd.get_dummies()`

.

3. Split the data 80/20 using `train_test_split`

.

4. Initialize the `RandomForestRegressor`

with `n_estimators=100`

and `max_depth=10`

to prevent overfitting.

5. Fit the model and evaluate using `mean_absolute_error`

.

``` python
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

# Initialize the model
rf_model = RandomForestRegressor(n_estimators=100, random_state=42)

# Training
rf_model.fit(X_train, y_train)

# Prediction
predictions = rf_model.predict(X_test)
print(f"MAE: {mean_absolute_error(y_test, predictions)}")
```

It's a good reminder that "simpler" isn't always "better" if the underlying data distribution is chaotic.

[Next Who actually has enough VRAM to run Qwen3.8-2. →](/en/threads/6061/)