What Makes a Best-Selling Novel? A Machine Learning Approach (2016)

A 2016 analysis of 17,744 Wikipedia plot summaries found that a logistic regression model using TF-IDF features could not predict best-seller status for any novel, with the highest probability at 0.39 for James Patterson's 'Cross Fire'. The study, based on data from Wikipedia and the New York Times Fiction Best Seller list, identified 'lawyer' and 'kill' as positive predictors and 'planet' as a negative predictor, but concluded that plot alone is insufficient to forecast commercial success.

A Machine Learning Approach In 2013, Ashok et al. http://aclweb.org/anthology/D/D13/D13-1181.pdf answered this question basing on the writing style, with 61–84% accuracy. This post, on the other hand, examines plot themes in best sellers. Note that my model can hardly predict the commercial success of a novel from its plot. That would be quite a surprising feat, making reviewers obsolete. My goal was more modest: finding statistically profitable topics to write about. Using PetScan https://petscan.wmflabs.org/ and Wikipedia’s page export https://en.wikipedia.org/wiki/Special:Export/ , I downloaded 25,359 Wikipedia articles belonging to Category:Novels by year https://en.wikipedia.org/wiki/Category:Novels by year . From each article, I extracted the section named Plot , Plot summary , Synopsis , etc. if present and, stripped of MediaWiki markup, saved it into an SQLite database along with the title of the novel, its year of publication, and a Boolean that indicates if it ever topped the New York Times Fiction Best Seller list https://en.wikipedia.org/wiki/Lists of The New York Times Fiction Best Sellers : SELECT title, year, was bestseller, length plot FROM Novels ORDER BY random LIMIT 5; Sharpe's Havoc | 2003 | 0 | 2759 The Rescue Sparks novel | 2000 | 1 | Slayers | 1989 | 0 | The Warden | 1855 | 0 | 2793 The Fourth Protocol | 1984 | 1 | 5666 SELECT count FROM Novels WHERE plot IS NOT NULL; 17744 SELECT count FROM Novels WHERE plot IS NOT NULL AND was bestseller; 398 SELECT min year FROM Novels -- The year of publication. WHERE was bestseller; -- The NYT list starts in 1942. 1941 To obtain easy to interpret results, I have built a logistic regression model on top of the TF–IDF https://en.wikipedia.org/wiki/Tf%E2%80%93idf transformation of articles processed by the Porter stemmer http://tartarus.org/martin/PorterStemmer/ . The parameters have default values. In particular, the logistic regression uses L2 regularization so all lowercase words that are not stopwords appear in the model. python import nltk from nltk.corpus import stopwords from nltk.stem import porter from sklearn import cross validation from sklearn import linear model from sklearn import pipeline from sklearn.feature extraction import text def Tokenize text, stemmer=porter.PorterStemmer , uppercase=set string.uppercase , stop set=set stopwords.words 'english' , punctuation re = re.compile ur' ’“”…–— " $%&\' +,\-./:;?@\ \\\ ^ {|}~ ', re.UNICODE : text = punctuation re.sub ' ', text tokens = nltk.word tokenize text return stemmer.stem x for x in tokens if x.lower not in stop set and x 0 not in uppercase X = y = connection = sqlite3.connect 'novels.sqlite' for row in connection.cursor .execute """SELECT plot, was bestseller FROM Novels WHERE year = 1941 AND plot IS NOT NULL""" : X.append row 0 y.append row 1 connection.close X train, X test, y train, y test = cross validation.train test split X, y, test size=0.3 model = pipeline.Pipeline 'tfidf', text.TfidfVectorizer lowercase=False, tokenizer=Tokenize , 'logistic', linear model.LogisticRegression model.fit X train, y train The model can return the probability of being a best seller for any novel b with a plot summary: logit b = −4.6 + 2.5 tfidf lawyer, b + 2.4 tfidf kill, b + ⋯ − 1.5 tfidf planet, b Pr was bestseller b |plot b = e logit b / 1 + e logit b To put these coefficients in context, tfidf lawyer, The Firm ≈ 0.06. As it happens, the model returns logit b 0, that is Pr was bestseller b |plot b 1/2 for no novel b from the train or test set. The highest probability, 0.39, is predicted for Cross Fire https://en.wikipedia.org/wiki/Cross Fire %28novel%29 , indeed a best seller in December 2010. Only if I disable the normalization in TF–IDF or weaken the regularization in the logistic regression, I can overfit the model to the train set while for the test set both its precision and recall would be at most 20%. But, like I wrote in the introduction, this is not the point of this exercise. Let us look at the words with high absolute value of coefficients. - Apparently, it pays off to write legal thrillers: lawyer +2.5, case +2.4, law +1.5, client +1.3, jury +1.3, trial +1.3, attorney +1.0, suspect +1.0, judge +0.9, convict +0.8; - kill +2.4, murder +1.8, terrorist +1.2, shoot +1.1, body +1.1, die +1.0, serial +0.9, attack +0.9, assassin +0.8, kidnap +0.8, killer +0.8. - Political thrillers are not bad either: agent +1.4, politics +1.4, president +1.3, defector +1.2. - Business may be involved: firm +1.3, company +1.3, career +1.1, million +1.0, success +1.0, business +0.9, money +0.9. - Finally, the characters should have families: husband +1.4, family +1.3, house +1.2, couple +1.2, daughter +1.2, baby +1.1, wife +1.0, father +1.0, child +0.9, birth +0.8, pregnant +0.8, and use a car +1.5 and a phone +0.8. The genres to avoid for prospective best-selling authors? - Sci-fi: planet −1.5, human −1.0, space −0.7, star −0.4, robot −0.3, orbit −0.3. - Children’s literature: boy −1.3, school −1.0, young −0.8, girl −0.8, youth −0.4, teacher −0.4, aunt −0.4, grow −0.4. - Geography and travels: village −1.0, city −1.0, ship −0.8, way −0.7, go −0.7, land −0.6, adventure −0.6, colony −0.5, native −0.5, follow −0.5, mountain −0.5, crew −0.5, forest −0.5, travel −0.5, inhabit −0.4, sail −0.4, road −0.4, map −0.3, tribe −0.3. - War: fight −1.0, warrior −0.6, war −0.6, weapon −0.5, soldier −0.5, army −0.5, ally −0.4, enemy −0.3, conquer −0.3. - Fantasy: magic −0.9, creature −0.5, magician −0.4, zombie −0.3, treasure −0.3, dragon −0.3. - History: princess −0.5, rule −0.5, kingdom −0.4, castle −0.4, century −0.4, ruler −0.3, palace −0.3 for what it’s worth, A Game of Thrones only made it to the third place on the list http://www.nytimes.com/best-sellers-books/2011-07-10/combined-print-fiction/list.html so it does not count as a best seller . Note that the code above ignores capitalized words. If it does not, the most significant words become the names of characters from best selling book series: Scarpetta https://en.wikipedia.org/wiki/Kay Scarpetta +3.0, Stephanie https://en.wikipedia.org/wiki/Stephanie Plum +2.9, Ayla https://en.wikipedia.org/wiki/Ayla %28Earth's Children%29 +2.0, etc., with additional insights like FBI +1.3, CIA +1.3, NATO +0.9, Soviet +0.9, or Earth −1.1.