Limits of the Bing, AFINN, and NRC Lexicons with the Tidytext Package in R

I am currently working on my conference paper, “Digital Analysis of Dickensian Characters and the Language of Emotion,” which I am supposed to present at the 2018 Dicken Society Symposium in Tübigen, Germany this summer. This paper concerns performing sentiment analysis using a computer programming tool. I decided to use R programming for sentiment analysis, because R provided the Tidytext and Syuzhet[i] packages for sentiment analysis based on “AFINN,” “Bing,” and “NRC.”[ii]

The Tidytext package draws upon three main lexicons for sentiment analysis: “Bing,” “AFINN,” and “NRC.” The Bing lexicon uses a binary categorization model that sorts words into positive or negative positions.[iii] The AFINN lexicon grades words between -5 and 5 (positive scores indicate positive sentiments).[iv] The NRC lexicon categorizes sentiment words into positive, negative, anger, anticipation, disgust, fear, joy, sadness, surprise and trust.[v]

My research mostly focuses on what words Dickens used, rather than the relationship of the words in sentences. This makes it less dangerous to analyze literary texts by drawing upon a big data tool, because I can shirk the drawbacks of text mining packages. Nonetheless, while I was working on the sentiment analysis of Dickens’ works, I found limitations in the Tidytext package for sentiment analysis, similar to how Joanna Swafford discovered drawbacks in the Syuzhet package when analyzing literary texts as sentences, since both the Tidytext and Syuzhet packages use the “Bing,” “AFINN,” and “NRC” lexicons.

  1. Classification of gender and appellation.

Miss_Dorrit1

The Bing and AFINN lexicons perceive the word “miss” as a negative word, considering that the word can be used as a verb. However, the NRC lexicon does not classify “miss” as a negative word. If “Miss. Amy” was classified as a negative word in Little Dorrit, the sentiment analysis would be inaccurate. If the analysis was solely based on word counts, gender classification would not matter. The most frequent word in many novels is “miss” because of the appellation, “Miss.” This is a problem in the tidytext package, rather than in the lexicons. The tidytext package needs to develop a function for classifying gender.

Little_Dorrit_Top10_Senti_Words1

  1. The classification of the NRC lexicon.

Screen Shot 2018-02-10 at 12.39.14 AM

The words in the NRC lexicon are often duplicated. As you can see in the chart above, “abandon” is contained in three categories, and “abandoned” in four. This duplication often results in difficulties during sentiment analysis. In the sentiment analysis chart for Dickens’ Little Dorrit, according to the NRC lexicon, “mother” ranks number 1 in “joy,” “negative,” and “sadness” categories, whereas in the Bing and AFINN lexicons, “mother” is not classified as an emotional word. The word “mother” should not be considered  an emotional word, but the NRC lexicon identifies it as so.

  1. The classification of human names in the NRC lexicon.

Human names should not be classified as emotional words. Surprisingly, I found “John” is classified as a negative word in the NRC lexicon; It is filtered in the Bing and AFINN lexicons.

The three lexicons were created by different developer teams, each aiming for sentiment analysis of all texts. Since the lexicons are not tailored to literary analysis, there are some issues for the sentiment analysis of literary texts, especially for the NRC lexicon, which seems to have been developed separately from the Bing and AFINN lexicons. For Digital Humanists using the three lexicons at the same time in their research, maintaining uniformity in their research methodology could become problematic.

Since the Tidytext package shares the three lexicons “Bing,” “AFINN,” and “NRC” with the Syuzhet package, both packages have the same limitation for sentiment analysis. So far, there are very few options to choose from for sentiment analysis. In order to ameliorate the issues, we need to tailor the algorithms of the code to the purpose of our research. To enhance the field of sentiment analysis, these packages should be routinely updated to implement improvements based on feedback from experts like Swafford.

 

References

Bing Liu, Minqing Hu and Junsheng Cheng. “Opinion Observer: Analyzing and Comparing Opinions on the Web.” Proceedings of the 14th International World Wide Web conference (WWW-2005), May 10-14, 2005, Chiba, Japan.

Finn Arup Nielsen. “A new ANEW: Evaluation of a word list for sentiment analysis in microblogs,” Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’:Big things come in small packages 718 in CEUR Workshop Proceedings : 93-98. 2011 May. http://arxiv.org/abs/1103.2903. See: http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010

Minqing Hu and Bing Liu. “Mining and Summarizing Customer Reviews.” Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2004), Aug 22-25, 2004, Seattle, Washington, USA. See: http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html#lexicon

Paracchini, Pier Lorenzo. “NLP – Sentiment Analysis using the tidytext package.” Dec 17, 2016. See: https://rstudio-pubs-static.s3.amazonaws.com/236096_2ef4566f995e48c1964013310bf197f1.html

Saif Mohammad and Peter Turney. “Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon.” In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, June 2010, LA, California. See: http://saifmohammad.com/WebPages/lexicons.html

Swafford, Joanna. “Problems with the Syuzhet Package.” Anglophile in Academia: Annie Swafford’s Blog, March 2, 2015. See: https://annieswafford.wordpress.com/2015/03/02/syuzhet/

 

[i] The Syuzhet package is a custom sentiment dictionary developed by the Nebraska Literary Lab.

[ii] There is the loughran lexicon too, but it is best suited for financial texts.

[iii] The Bing lexicon was created by Bing Liu and collaborators.

[iv] The AFINN lexicon was made by Finn Årup Nielsen.

[v] The NRC lexicon was announced by Saif Mohammad and Peter Turney. NRC stands for National Research Council in Canada. Check the website: http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm

Advertisements

2 thoughts on “Limits of the Bing, AFINN, and NRC Lexicons with the Tidytext Package in R

  1. FYI, ‘John’ is probably negative because it’s an English euphemism for the toilet. You would need NER to disambiguate, especially if you’ve case-folded the original text.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s