In 2013, Ashok et al. answered this question basing on the writing style, with 61–84% accuracy. This post, on the other hand, examines plot themes in best sellers. Note that my model can hardly predict the commercial success of a novel from its plot. That would be quite a surprising feat, making reviewers obsolete. My goal was more modest: finding statistically profitable topics to write about.
Using PetScan and Wikipedia’s page export, I downloaded 25,359 Wikipedia articles belonging to Category:Novels by year. From each article, I extracted the section named Plot, Plot summary, Synopsis, etc. if present and, stripped of MediaWiki markup, saved it into an SQLite database along with the title of the novel, its year of publication, and a Boolean that indicates if it ever topped the New York Times Fiction Best Seller list:
SELECT title, year, was_bestseller, length(plot) FROM Novels
ORDER BY random() LIMIT 5;
Sharpe's Havoc | 2003 | 0 | 2759
The Rescue (Sparks novel) | 2000 | 1 |
Slayers | 1989 | 0 |
The Warden | 1855 | 0 | 2793
The Fourth Protocol | 1984 | 1 | 5666
SELECT count(*) FROM Novels
WHERE plot IS NOT NULL;
17744
SELECT count(*) FROM Novels
WHERE plot IS NOT NULL AND was_bestseller;
398
SELECT min(year) FROM Novels -- The year of publication.
WHERE was_bestseller; -- The NYT list starts in 1942.
1941
To obtain easy to interpret results, I have built a logistic regression model on top of the TF–IDF transformation of articles processed by the Porter stemmer. The parameters have default values. In particular, the logistic regression uses L2 regularization so all lowercase words that are not stopwords appear in the model.
import nltk
from nltk.corpus import stopwords
from nltk.stem import porter
from sklearn import cross_validation
from sklearn import linear_model
from sklearn import pipeline
from sklearn.feature_extraction import text
def Tokenize(
text,
stemmer=porter.PorterStemmer(),
uppercase=set(string.uppercase),
stop_set=set(stopwords.words('english')),
punctuation_re = re.compile(
ur'[’“”…–—!"#$%&\'()*+,\-./:;?@\[\\\]^_`{|}~]',
re.UNICODE)):
text = punctuation_re.sub(' ', text)
tokens = nltk.word_tokenize(text)
return [stemmer.stem(x) for x in tokens
if x.lower() not in stop_set and x[0] not in uppercase]
X = []
y = []
connection = sqlite3.connect('novels.sqlite')
for row in connection.cursor().execute(
"""SELECT plot, was_bestseller FROM Novels
WHERE year >= 1941 AND plot IS NOT NULL"""):
X.append(row[0])
y.append(row[1])
connection.close()
X_train, X_test, y_train, y_test = (
cross_validation.train_test_split(X, y, test_size=0.3))
model = pipeline.Pipeline(
[('tfidf', text.TfidfVectorizer(
lowercase=False, tokenizer=Tokenize)),
('logistic', linear_model.LogisticRegression())])
model.fit(X_train, y_train)
The model can return the probability of being a best seller for any novel b with a plot summary:
logit(b) = −4.6 + 2.5 tfidf(lawyer, b) + 2.4 tfidf(kill, b) + ⋯ − 1.5 tfidf(planet, b)
Pr(was_bestseller(b)|plot(b)) = elogit( b) / (1 +
e
logit()
b)To put these coefficients in context, tfidf(lawyer, The Firm) ≈ 0.06. As it happens, the model returns logit(b) > 0, that is Pr(was_bestseller(b)|plot(b)) > 1/2 for no novel b from the train or test set. The highest probability, 0.39, is predicted for Cross Fire, indeed a best seller in December 2010. Only if I disable the normalization in TF–IDF or weaken the regularization in the logistic regression, I can overfit the model to the train set while for the test set both its precision and recall would be at most 20%. But, like I wrote in the introduction, this is not the point of this exercise. Let us look at the words with high absolute value of coefficients.
- Apparently, it pays off to write legal thrillers: lawyer +2.5, case +2.4, law +1.5, client +1.3, jury +1.3, trial +1.3, attorney +1.0, suspect +1.0, judge +0.9, convict +0.8;
- kill +2.4, murder +1.8, terrorist +1.2, shoot +1.1, body +1.1, die +1.0, serial +0.9, attack +0.9, assassin +0.8, kidnap +0.8, killer +0.8.
- Political thrillers are not bad either: agent +1.4, politics +1.4, president +1.3, defector +1.2.
- Business may be involved: firm +1.3, company +1.3, career +1.1, million +1.0, success +1.0, business +0.9, money +0.9.
- Finally, the characters should have families: husband +1.4, family +1.3, house +1.2, couple +1.2, daughter +1.2, baby +1.1, wife +1.0, father +1.0, child +0.9, birth +0.8, pregnant +0.8, and use a car +1.5 and a phone +0.8.
The genres to avoid for prospective best-selling authors?
- Sci-fi: planet −1.5, human −1.0, space −0.7, star −0.4, robot −0.3, orbit −0.3.
- Children’s literature: boy −1.3, school −1.0, young −0.8, girl −0.8, youth −0.4, teacher −0.4, aunt −0.4, grow −0.4.
- Geography and travels: village −1.0, city −1.0, ship −0.8, way −0.7, go −0.7, land −0.6, adventure −0.6, colony −0.5, native −0.5, follow −0.5, mountain −0.5, crew −0.5, forest −0.5, travel −0.5, inhabit −0.4, sail −0.4, road −0.4, map −0.3, tribe −0.3.
- War: fight −1.0, warrior −0.6, war −0.6, weapon −0.5, soldier −0.5, army −0.5, ally −0.4, enemy −0.3, conquer −0.3.
- Fantasy: magic −0.9, creature −0.5, magician −0.4, zombie −0.3, treasure −0.3, dragon −0.3.
- History: princess −0.5, rule −0.5, kingdom −0.4, castle −0.4, century −0.4, ruler −0.3, palace −0.3 (for what it’s worth, A Game of Thronesonly made itto the third place on the listso it does not count as a best seller).
Note that the code above ignores capitalized words. If it does not, the most significant words become the names of characters from best selling book series: Scarpetta +3.0, Stephanie +2.9, Ayla +2.0, etc., with additional insights like FBI +1.3, CIA +1.3, NATO +0.9, Soviet +0.9, or Earth −1.1.