Empirical Methods

The idea of exploring questions about literature empirically can be one of the most exciting but also the most daunting things about getting to grips with cognitive approaches. But there are as many methods as there are questions to ask, and many of them can be tried out in a modest way without too long a learning curve or substantial an investment of resources. This section provides an overview of some commonly employed methods for studying both texts and readers, and the kinds of specific questions they can be used to tackle.

Compiled by Emily Troscianko

Hidden accordion
Why do it?

The most basic argument for working empirically is that putting our theories and arguments to some kind of empirical test is an important way of corroborating, falsifying, and refining them. Here are some general introductions to some reasons and methods for studying literature empirically:

Bortolussi, M., and Dixon, P. (2003). Preliminaries. In M. Bortolussi and P. Dixon, Psychonarratology: Foundations for the empirical study of literary response, pp. 34-59. Cambridge: Cambridge University Press.

‘This chapter was designed in part to address the needs, interests, and concerns of literary scholars who may be intrigued by the empirical study of literary response but lack the confidence to pursue it on their own.’ The chapter includes an introduction to psychonarratology (the study of the psychological processing of narrative form) and to the concepts of the ‘statistical reader’ and ‘measurement distributions’ of particular variables within a given population. It sets out some of the epistemological assumptions involved in empirical research on literature, as well as arguments for the importance of distinguishing clearly between text features and reader constructions, and the value of carrying out controlled ‘textual experiments’ in which texts are manipulated and changes in readers’ responses observed so as to eliminate potential confounds and move closer to causal explanations. (There are also some brief remarks on significance testing.) Subsequent chapters go into detail on the topics of narrator, events and plot, characters and characterisation, perception and focalisation, and represented speech and thought, in each case clarifying what it means to take a psychonarratological approach, and presenting existing empirical evidence on reader responses.

Hanauer, D. I., Kuiken, D., and Hakemulder, F. (2013). The scope of SSOL: A discussion of the boundaries of science and literature. Scientific Study of Literature, 3(2), 169-174.

The editors of a then two-year-old journal try to clarify what they mean by science and by literature.

Martindale, C. (1996). Empirical questions deserve empirical answers. Philosophy and Literature, 20, 347-361.

A forceful early advocacy of empirical literary studies, focused on the question of how much consistency or variation there is in trained and untrained readers’ interpretations and classifications of literature and other media. The sentence ‘Theories imply hypotheses, and hypotheses imply empirical or experimental testing’ nicely sums it up.

Steen, G. J. (1991). The empirical study of literary reading: Methods of data collection. Poetics, 20, 339-575.

An outline of many of the most common measures used in experiments with literary texts, including verbal and non-verbal, with varying degrees of control imposed by the experimenter, and administered before, during, or after reading. Each strikes at a different point the balance between analysability and manipulability on the one hand, and natural richness and validity on the other.

The future of scientific studies in literature. Special issue, Scientific Study of Literature, 1(1).

The contributions to SSOL’s first issue include a range of theoretically and empirically orientated discussions, on topics including science and literariness, empirical narratives and ‘symptoms of science’, literature as entertainment, and cultural colonialism; the importance of interdisciplinarity in the teaching of literature; corpus and computational linguistics, the use of ‘textoids’ or more naturalistic texts in experimental work, and individual differences and commonalities amongst readers and uses of language.

Rating scales and questionnaires

Some variant on rating-scale research is the most common way of working empirically with literary response. You can make up your own questions, or use sets of questions tested by other people, or a mixture.

Validated literature/narrative-specific questionnaires

Busselle, R., and Bilandzic, H. (2009). Measuring narrative engagement. Media Psychology, 12, 321-347.

Describes the development and validation of a scale to measure narrative engagement based on a mental-models approach to narrative processing across media. The scale distinguishes between narrative understanding, attentional focus, emotional engagement, and narrative presence, and is validated using data from viewers of movies and TV programmes in different viewing situations and from the USA and Germany.

Green, M. C., and Brock, T. C. (2000). The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology, 79(5), 701-721. http://www.communicationcache.com/uploads/1/0/8/8/10887248/the_role_of_transportation_in_the_persuasiveness_of_public_narratives.pdf

Develops and validates a transportation scale to measure absorption into a story (conceived of as involving mental imagery, emotion, and attentional focus). Experiments using the scale are reported that show effects of transportation level on story-consistent beliefs, and on evaluations of protagonists and other textual features, while finding no effect on transportation from labelling stories as fact or as fiction.

(For an application of this scale in two experiments investigating interactions between readers’ pre-reading emotional states and the emotional tone of the narrative, see also Green, M. C., Chatham, C., and Sestir, M. A. (2012). Emotion and transportation into fact and fiction. Scientific Study of Literature, 2(1), 37-59. See also Thompson and Haddock 2012 (in the Cognitive Humanities bibliography section Personality and Individual Difference) for use of the transportation scale in the context of a study on individuals’ drinking habits, attitudes, and intentions.)

Kuijpers, M. M., Hakemulder, F., Tan, E. S., and Doicaru, M. M. (2014). Measuring absorbing reading experiences: Developing and validating a self-report scale to measure story world absorption. Scientific Study of Literature, 4(1), 89-122.

The scale includes the dimensions of attention, transportation, emotional engagement, and mental imagery (these subscales can also be used independently), and predicts two distinct evaluative responses: enjoyment and impact. The authors argue that the subscale of narrative presence in Busselle and Bilanszic’s narrative engagement scale confounds two dimensions (transportation and attention), and that Green and Brock’s transportation scale also lacks precision relative to theirs.

Kuiken, D., Campbell, P., and Sopcak, P. (2012). The Experiencing Questionnaire. Scientific Study of Literature, 2(2), 243-272.

Develops a 58-item questionnaire to assess ‘some relatively uncommon but theoretically significant types of reading experience’ associated with generative and self-altering qualities, with a phenomenological basis. The EQ scales are shown to differentiate between theoretically relevant orientations towards literature – objective and subjective, secular and spiritual – and, in combination, to reflect ‘apex moments’ of ‘sublime disquietude’ or ‘sublime enthrallment’.

Miall, D.S., and Kuiken, D. (1995). Aspects of literary response: A new questionnaire. Research in the Teaching of English, 17, 37-58. https://sites.ualberta.ca/~dmiall/reading/LRQ_95.htm

The Literary Response Questionnaire provides scales to measure insight, empathy, imagery vividness, leisure escape, concern with author, story-driven reading, and rejection of literary values in readers’ orientations towards literary texts. This paper describes the LRQ’s development and relates its subscales to readers’ personality traits and learning skills.

Ad hoc scales

Carney, J., Wlodarski, R., and Dunbar, R. (2014). Inference or enaction? The impact of genre on the narrative processing of other minds. PLoS One, 9(12), e114172. http://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0114172

Administers rating scales tapping engagement with the text and evaluations of literary quality to investigate whether narratives affect how we interact with other minds. A contrast was found between how the literary quality of relationship and espionage stories (as examples of evolutionarily familiar versus unfamiliar scenarios) was assessed, depending on how many levels of intentionality they incorporated.

Hilscher, M. C., and Cupchik, G. C. (2005). Reading, hearing, and seeing poetry performed. Empirical Studies of the Arts, 23(1), 47-64.

Investigates different ways of experiencing poetry by means of two questionnaires constructed for the purposes of this study: the General Poetry Questionnaire (about general experiences and impressions of poetry, administered beforehand) and the Poetry Reception Questionnaire (about the cognitive-emotional nuances of poetry reception, administered after reading), plus one open-ended question about the poem’s meaning (answered before the PRQ). The free-response paragraphs were analysed using a qualitative method of category construction. The study found that people prefer reading poetry rather than hearing it read or seeing it performed, since this lets them explore the text more independently and creatively.

Online surveys

Koopman, E. (2011). Predictors of insight and catharsis among readers who use literature as a coping strategy. Scientific Study of Literature, 1(2), 241-259. https://benjamins.com/#catalog/journals/ssol.1.2.04koo/details

Uses an online survey to ask respondents to report on music and literature that helped them get through a difficult time in their lives. They were asked questions (on 7-point scales) about their loss experience, coping style, and engagement with the music or literature. Aesthetic feelings (attention to and appreciation of stylistic features) were found to correlate with absorption and with experiencing more thoughts during reading, but not with catharsis or insight, which correlated with narrative feelings (identifying with the character and feeling absorbed in the narrative world). However, a subgroup also found therapeutic value in the comfort of aesthetic beauty.

Qualitative methods

There are a variety of responses to the restrictions imposed by rating-scale paradigms; here are just a couple.

Dixon, P., Bortolussi, M., and Mullins, B. (2015). Judging a book by its cover. Scientific Study of Literature, 5(1), 23-48.

In this study, self-identified science-fiction fans and mystery fans sorted 80 randomly selected book genres from both genres into groups of their own devising; their sorts were used to identify similarity among books, and that similarity structure was used to measure similarity among participants, with cluster analysis used to find groups who sorted similarly. Group membership was related to reported knowledge about the genres, indicating the effectiveness of covers as an implicit signalling system between publishers and experienced readers of a given genre.

Van Peer, W. (1990). The measurement of metre: Its cognitive and affective functions. Poetics, 19, 259-275.

Participants’ responses to a metrical and a non-metrical version of a poem are measured using 16 semantic differential scales (pairs of adjectives chosen to tap aesthetic reactions) plus a multiple-choice recall task and some comprehension questions. Metrical structure was found to enhance aesthetic pleasure and the ability to recognise segments of the text afterwards.

(For a review of the semantic differential method, see also Heise, D. R. (1970). The semantic differential and attitude research, in G. F. Summers (Ed), Attitude measurement (pp. 235-253). Chicago: Rand McNally. http://www.indiana.edu/~socpsy/papers/AttMeasure/attitude..htm.)

Open-ended questions and free-response/think-aloud protocols

Another simple way to avoid the limitations of rating scales is to ask open-ended questions. This can also be done in a real-time fashion to avoid the drawbacks of only asking people about their experience once it’s over. (The only difficulty then is how to analyse the reams of raw data you end up with.)

Claassen, E. (2012). Author inferences in thinking aloud, in E. Claassen, Author representations in literary reading (pp. 61-101). Amsterdam: John Benjamins.

Asks whether and how readers generate inferences about authors during the reading of narrative text, and if so, whether they can be revealed through think-aloud protocols. Includes methodological reflections on think-aloud as opposed to other methods, and what they can and cannot be used to demonstrate. In particular, the exploratory study described investigates whether inferences about authors contribute to a mental representation of the communication context, and whether a narrator’s visibility or particular reading strategies affect these inferences. Participants were asked to share their thoughts when they reached a black mark in the text, and also to give a summary of the text afterwards (with or without an instruction to reflect on authorial intention), as well as completing a short questionnaire on text evaluation and reading behaviour. The protocol coding procedure and statistical (chi-squared) analysis are also described in detail, and the findings include reflections on the limitations of the think-aloud method when it comes to automatically generated inferences.

Gibbs, R. W., and Blackwell, N. (2012). Climbing the ladder to literary Heaven: A case study of allegorical interpretation. Scientific Study of Literature, 2(2), 199-217.

Asked participants to read a passage of extended metaphor from a novel and immediately write out their responses to a series of questions like ‘Please describe what the infinitely tall ladder refers to or represents’, ‘What would happen if the author “loosened his grip” while on the ladder and “fell to one side”?’, and ‘Describe the bodily sensations you felt while reading the story’. Participants’ responses manifested common features of the LIFE IS A JOURNEY metaphorical field and also elaborated on it with personal readings.

Miall, D. S., and Kuiken, D. (2001). Shifting perspectives: Readers’ feelings and literary response. In W. van Peer and S. Chatman (Eds), New perspectives on narrative perspective, (pp. 289-302). Albany, NY: State University of New York Press.

Investigates the relationships between aesthetic feeling, foregrounding, and reader perspective using a combination of measures: pre-validated questionnaires (the Literary Response Questionnaire and the Multidimensional Personality Questionnaire), reading times, and think-aloud protocols (relating to thoughts and feelings while reading), plus pre-existing and tailored discourse measures (for propositional features, discontinuities, and experiential perspective). The think-aloud protocols were only analysed informally, to add depth to the results gathered using the other measures. The findings contribute to a model of the interpretive process as a phasic cycle.

Probes during reading

The trouble with think-aloud methods is how severely they disrupt the ‘normal reading experience’. Other kinds of real-time probe may be much less intrusive – they seem to be being developed primarily for studying film, but would transfer well to literary studies.

Bezdek, M., and Gerrig, R. (2016). When narrative transportation narrows attention: Changes in attentional focus during suspenseful viewing. Media Psychology online first. https://www.researchgate.net/profile/Matthew_Bezdek/publication/293174035_When_Narrative_Transportation_Narrows_Attention_Changes_in_Attentional_Focus_During_Suspenseful_Film_Viewing/links/56b61a1b08ae44bb330783d1.pdf

In order to explore the dynamic role of attention in narrative transportation, a reaction time-based method of real-time probing is developed to indicate the extent to which the primary task (watching a movie excerpt) commands viewers’ cognitive resources: participants were asked to respond to a probe tone by pressing a button on a computer keyboard. Of particular interest were ‘hot spot’ moments where potential negative outcomes are emphasised. Reaction times were found to be longer, and more probes were missed, during hot spots than during cold spots. The researchers also administered a visual recognition memory task, asking participants to identify still images from the film clips, though confounding factors were identified for this measure, and they concluded that measuring narrative recall may work better.

Troscianko, T., Meese, T., and Hinde, S. (2012). Perception while watching movies: Effects of physical screen size and scene type. i-Perception, 3, 414-425. https://core.ac.uk/download/pdf/9624299.pdf?repositoryId=7

Develops a simple measure to track ongoing ‘presence’ (involvement) in real time: participants were prompted to report their level of presence using a line-bisection task at intervals in a 45-minute section of film (‘You should make a mark on the line to indicate how “present” you feel in the movie just before the light came on. If you feel completely “in the story,” then your mark should be at the far right of the line. If you feel that you are viewing the movie, then place your mark on the far left of the line.’). Measures of pupil dilation and reaction times (a timed button pressed signalled by a beep) were also obtained. The first study found a correlation between presence and physical screen size, and presence and scenes focused on faces rather than landscapes. The second study found correlation between presence and pupil dilation (even when controlling for variations in screen luminance), though not between presence and reaction times (but overall presence levels were lower in this experiment, presumably due to increased intrusion from the three measures), suggesting that pupil dilation may be a more sensitive measure than reaction times.

Eye tracking

Eye tracking is a way of gathering lots of detailed information about cognitive processing indirectly (i.e. without needing to ask people to report on anything), and as the technology advances, this method also becomes ever less intrusive.

Rayner, K. (1998). Movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422.

A review of the specifics of how eye movements reflect moment-to-moment cognitive processes involved in reading. The paper includes an outline of the basic characteristics of eye movements and fixations, some methods for measuring them, and features of eye-movement patterns like regression, refixations, and word skipping, as well as discussion of variation in reading practices during typical development and in people with reading difficulties.

Kaakinen, J. K., and Hyönä, J. (2008). Perspective-driven text comprehension. Applied Cognitive Psychology, 22, 319-334.

The texts used here are in no sense literary, but the study illustrates a combination of eye tracking with written free recall of a text, following instructions to adopt a particular perspective when reading a text version where the (ir)relevance of the text to the perspective was either transparent or opaque. The authors conclude that perspective-related prior knowledge modulates the perspective effects observed in text processing, and that signalling of (ir)relevance helps readers encode relevant information to memory.

Koops van ‘t Jagt, R., Hoeks, J., Dorleijn, G., and Hendriks, P. (2014). Look before you leap: How enjambment affects the processing of poetry. Scientific Study of Literature, 4(1), 3-24.

Uses eye tracking to investigate the differences in reading poetry with or without enjambements (of both the prospective and the retrospective kind), using both authentic and specially constructed examples of enjambement. The study found significant differences between conditions, favouring a dynamic model of integrative language processing.

Brain imaging

Certainly the trendiest way of doing cognitive research right now, if not the most informative, brain imaging has a solid history in the study of metaphor processing, and is now making its way into the study of specifically literary reading.

Mashal, N., Faust, M., Hendler, T., and Jung-Beeman, M. (2007). An fMRI investigation of the neural correlates underlying the processing of novel metaphoric expressions. Brain and Language, 100, 115-126.

One of many fMRI studies on the processing of literal versus metaphorical language (often also using irony as an additional control) that demonstrate the involvement of the right hemisphere (usually associated with visuospatial rather than linguistic processing) in metaphor processing, this one suggests that interpretive salience – or the extent to which a metaphor is novel (nonsalient) versus conventional (salient) – is actually the primary factor predicting RH involvement.

Miall, D. S. (2011). Emotions and the structuring of narrative experience. Poetics Today, 32, 323-248.

Argues that findings from studies of evoked-response potentials (ERPs – an electrical potential recorded using EEG or EMG) indicate an early (within the first half second of response) and central role for emotion in the cognitive processing of language, including in areas such as inferencing, autobiographical memory and self-reference, anticipation, narrativising, and empathy.

Phillips, N. M. (2015). Literary neuroscience and history of mind: An interdisciplinary fMRI study of attention and Jane Austen. In L. Zunshine (Ed.), The Oxford handbook of cognitive literary studies (pp. 55-81). New York: Oxford University Press.

Describes an experiment in which participants (18 PhD students expert in literary analysis) were asked to switch between close reading and reading for pleasure as they read Chapter 2 of Mansfield Park while lying in the fMRI scanner. Though analysis was still in progress when this was written, the study suggested widespread differences in activation patterns between the two. It also used fMRI-compatible eye tracking, and analysis of those data are forthcoming too.

Yarkoni, T., Speer, N. K., Balota, D. A., McAvoy, M. P., and Zacks, J. M. (2008). Pictures of a thousand words: Investigating the neural mechanisms of reading with extremely rapid eventrelated fMRI. NeuroImage, 42, 973-987.

Describes a new ‘event-related reading’ method for fMRI, tested in an experiment in which reading of coherent narrative revealed widespread effects of orthographic, phonological, contextual, and semantic variables on brain activation. Results appear to replicate across previous single-word fMRI experiments, and to predict individual differences in reading comprehension.

Other physiological measures

If you don’t have the cash or the inclination to go down the neuro route, there are other physiological measures that can serve as indirect indicators of particular types of response.

Ravja, N., Saari, T., Kallinen, K., and Laarni, J. (2006). The role of mood in the processing of media messages from a small screen: Effects on subjective and physiological responses. Media Psychology, 8(3), 239-265.

Facial EMG measures muscle activity in the face by detecting and amplifying the electrical pulses generated when muscle fibres contract; the focus is usually on two major muscles groups associated with frowning and smiling respectively. This study used facial electromyography (facial EMG) and cardiac heartbeat intervals as indicators of valence and arousal in the emotional reception of verbal versus video messages in different starting mood conditions and with different levels of relevance to the participants. Participants were also asked to rate their emotional reactions on a valence scale consisting of 9 pictures of human faces with expressions ranging from a severe frown to a broad smile. It found that higher relevance, arousal, and associated muscular activity were found for the verbal condition when in a depressed mood, and the reverse when in a joyful, relaxed, or playful mood.

Riese, Bayer, Lauer, and Schacht. (2014). Pupillary responses to suspense in literary classics. Scientific Study of Literature, 4(2), 211-232. http://gerhardlauer.de/publications/

Pupil dilation is well known to correlate with arousal (see also Troscianko et al., 2011), and also to be related to attention. This study investigated feelings of suspense using pupillometry while participants listened to recordings of passages from Fontane and Storm. Detailed suspense ratings had previously been obtained from expert and nonexpert readings using two different methods: applying an 11-point scale to each sentence in printed copies of the texts, or listening to the texts being read while noting the suspense value (also on a scale from 0 to 10) at the end of each line of the transcript, in both cases relying on subjective appraisal rather than technical reasoning. Significant correlations were found between pupil diameter and changing arc of suspense, offering evidence for the usefulness of this technique as an indicator of suspense.

Indirect behavioural measures

On the behavioural side of things, too, indirect measures can be useful, though the benefit of not needing to rely on self-report is countered by the lack of certainty that a given measure reflects the variable you think it does, and only that one.

Bryant, D. J., Tversky, B., and Franklin, N. (1992). Internal and external spatial frameworks for representing described scenes. Journal of Memory and Language, 31, 74-98.

Explored readers’ mental models of narrative scenes described from the perspective of either an observer within the scene, surrounded by objects, or one outside the scene, with objects in front (and spatial relations described relative to other objects not the observer). Participants were asked questions about the locations of objects, and their reaction times differed for the two conditions, with faster responses to questions of front than back in the internal condition (reflecting physiological constants) and no difference between front and back questions in the external condition (where the body axis is irrelevant), and responses faster overall to questions answered from an external perspective. Subsequent experiments configured descriptions from the perspective of a central person or inanimate object, leaving the reader freer to choose what perspective to adopt. This method offers a way of ascertaining the perspective readers adopt in response to a text where perspective is unspecified or complex.

Emmott, C., Sanford, A. J., and Dawydiak, E. J. (2007). Stylistics meets cognitive science: Studying style in fiction and readers’ attention from an interdisciplinary perspective. Style, 41(2), 204-224.

Outlines the principles and practice of ‘depth of processing’ testing, with literary relevance in the realm of foregrounding devices in particular. Traditional methods and the newer ‘change detection’ method are introduced, the latter adapted for verbal texts from vision research on the phenomenon of change blindness. A study is described to assess the effects of ‘attention-capturing’ devices at different linguistic and narrative levels, with a surprising finding that narrative foregrounding actually reduces change detection demanding further exploration.

Whalen, D. H., Zunshine, L., and Holquist, M. (2012). Theory of Mind and embedding of perspective: A psychological test of a literary ‘sweet spot’. Scientific Study of Literature, 2(2), 301-315. http://www.haskins.yale.edu/Reprints/HL1713.pdf

Reading times alongside comprehension questions of varying difficulty were used to test a hypothesis about theory of mind and literature: that three levels of perspective embedding might provide a helpful amount information about characters without overwhelming, and that this ‘sweet spot’ could be preferred in part thanks to ease of processing. A second study imposed a fixed reading speed based on the average reading times from the first experiment. Both found that degree of perspective embedding affected cognitive engagement, with zero embedding read slower than 1-3 levels, and about the same as 4 levels, while comprehension error increased only with 5 levels.

Zacks, J. M., Speer, N. K., and Reynolds, J. R. (2009). Segmentation in reading and film comprehension. Journal of Experimental Psychology: General, 138(2), 307-327.

Applied the reading-times method to test a hypothesis about how readers use natural boundaries in ongoing activity evoked in narrative film as an important part of comprehension. The set of experiments involved collecting reading times, segmentation judgements (press a button when you judge that one meaningful unit of activity has ended and another begun), and predictability ratings (also gathered during the viewing), plus cued recall questions to test comprehension for a film that had been pre-coded for situational changes. In this study, interactions between variables of situation change in the film and patterns of segmentation and reading times were analysed at an individual level, rather than just using group-averaged reading times; good agreement about event boundaries was found between individuals, but consistency was better within individuals.

Zwaan, R. A., Radvansky, G. A., Hilliard, A. E., and Curiel, J. M. (1998). Constructing multidimensional situation models during reading. Scientific Studies of Reading, 2(3), 199-220.

Used reading times to investigate which dimensions (time, space, physical and psychological causation, protagonist motivation, and new protagonists) of the mental model, or situation model, are monitored by readers of narrative, finding that the spatial dimension was the only one not monitored (i.e. spatial discontinuities did not lead to increased reading times). The spatial dimension was brought into line with the others in a second experiment where participants memorised a map of the building in which the described events took place, which encouraged them to monitor spatial continuity as they would otherwise not bother to do.

Content analysis / discourse analysis / stylometrics

In the final section, we turn to methods directed at the texts themselves, rather than at readers’ responses to them.

Software

Coh-Metrix

Funnels into a range of basic descriptive indices, plus outputs for dimensions including word concreteness, syntactic simplicity, referential cohesion (words and ideas that overlap across sentences and the entire text), deep cohesion (causal and intentional connectives), verb cohesion (overlapping verbs), connectivity (explicit conveying of logical connections), temporality, and narrativity), as well as measures for similarity between adjacent sentences, lexical diversity, situation model construction, syntactic complexity and pattern density, words before the main verb (an index of working memory load), word information, and readability.

Free online trial versions of the web tools plus extensive documentation are available here: http://cohmetrix.com. A free text analysis service for corpora over 15,000 words is also on offer.

LIWC (pronounced Luke): Linguistic Inquiry and Word Count dictionary

Contains 4,500 words and word stems, filed in one or more of around 80 outputs: 4 general descriptor categories (total word count, words per sentence, % of words captured by the dictionary, % of words longer than 6 letters), 22 standard linguistic dimensions (e.g. % of words that are pronouns, auxiliary verbs, etc.), 32 word categories tapping psychological constructs (e.g. affect, cognition, biological processes), 7 personal concern categories (e.g. work, home, leisure activities), 3 paralinguistic dimensions (assents, fillers, nonfluencies), and 12 punctuation categories. See http://www.liwc.net/LIWC2007LanguageManual.pdf for the development and psychometric properties, and http://liwc.wpengine.com to buy. The academic version currently costs £74.06 for an unlimited licence, or £8.20 for 30 days.

General advocacies

Graesser, A. C., Dowell, N., and Moldovan, C. (2011). A computer’s understanding of literature. Scientific Study of Literature, 1(1), 24-33. https://benjamins.com/ – catalog/journals/ssol.1.1.03gra/details

An introduction to how and why to use LIWC and Coh-Metrix to analyse literary texts.

Pennebaker, J. W., and Ireland, M.E. (2011). Using literature to understand authors: The case for computerized text analysis. Scientific Study of Literature, 1(1), 34-48.

Makes a general case for the importance of analysing ‘almost-invisible function words’ (pronouns, prepositions, etc.) rather than just ‘content’ words to learn about the social and psychological worlds of the authors who created them.

Quantitative methods

Nichols, R., Lynn, J., Purzycki, B. G. (2014). Toward a science of science fiction: Applying quantitative methods to genre individuation. Scientific Study of Literature, 4(1), 25-45.

To address the question of what genre is, and what distinguishes a genre like science fiction from other genres, this article presents a method of quantitative genre profiling that uses the word categories of the LIWC analyses to test a well-known literary theory of science fiction as offering particular representations of cognition and estrangement, as distinct from fantasy and mystery. Following presentation of the findings on similarities and differences between the three genres, the paper also includes a general discussion of the value of conducting quantitative empirical work on literature and of testing the hypotheses of literary theory.

Anderson, T., and Crossley, S. (2011) ‘Rue with a difference’: A computational stylistic analysis of the rhetoric of suicide in Hamlet’, in M. Ravassat and J. Culpeper (Eds), Stylistics and Shakespeare’s language: Transdisciplinary approaches, pp. 192-214. London: Continuum.

Demonstrates the complementarity of stylistic and literary interpretations using lexico-semantic and corpus approaches to analyse Hamlet’s and Ophelia’s dialogue for suicidal discourse, to reveal semantic prosodies and multiword meaning structures that have been overlooked by literary critics. Using the LIWC, and with Horatio’s and Laertes’s dialogue as controls, the authors construct and test specific linguistic hypotheses about suicidal rhetoric in the play.

Egbert, J. (2012). Style in nineteenth century fiction: A multi-dimensional analysis. Scientific Study of Literature, 2(2), 167-198.

This paper presents a study of a large, principled corpus of 19th-century fiction using a multidimensional corpus stylistics approach which aims to consider ‘the full set of core linguistic features’. The key dimensions of variation are interpreted as ‘thought presentation versus description’, ‘abstract exposition versus concrete action’, and ‘dialogue versus narrative’, and can be used to compare authorial styles and assess the level of variation within novels by the same author.

Zöllner, K. (1990). ‘Quotation analysis’ as a means of understanding comprehension processes of longer and more difficult texts. Poetics, 19, 293-322.

Quotation analysis is presented as a method for explaining and predicting which sections of well-known texts are quoted from and interpreted more frequently than others, which more generally can help us understand the semantic and structural makeup of ‘classical’ texts. The paper sets out the method used to code quotations and interpretations of VIPs (very important passages) from Gulliver’s Travels for analysis, and draws out points of interest such as the high polyvalence of VIPs and the purposes for which they are used, and how certain interpretations become classics in their own right.

Qualitative methods

Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. http://eprints.uwe.ac.uk/11735/2/

Introduces thematic analysis as an accessible and theoretically flexible way of analysing qualitative data, and clearly outlines the analytical process (including setting out the difference between codes and themes, and how to draw conclusions from the analysis) as well as pitfalls to avoid.

Schreier, M. (2012). Qualitative content analysis in practice. London: Sage.

Covers what qualitative content analysis is, how blurred the boundary is between it and its quantitative sister, and how the coding process works, from how to build and evaluate a coding frame to carrying out the main analysis and presenting the results. A lot of the key sections are available via Amazon preview when signed in: https://www.amazon.co.uk/Qualitative-Content-Analysis-Practice-Schreier/dp/1849205930

Other methods

A creative category in need of expansion; watch this space!

Yurievich Manin, D. (2012). The right word in the left place: Measuring lexical foregrounding in poetry and prose. Scientific Study of Literature, 2(2), 273-300.

Uses an online literary game where players guess words in fragments of real texts to quantify two aspects of lexical foregrounding – unpredictability and constrainedness (or irreplaceability) – as they help characterise the poetry/prose distinction and illuminate the formal constraints of different poetic forms.

Statistics

And last but not least, with all datasets comes the need to analyse them – not always using statistics, but often. Here are some relatively painless ways to get started.

Hoover, D. L. (2008). Quantitative analysis and literary studies. In S. Schreibman and R. Siemens (Eds), A companion to digital literary studies, Ch 28. Oxford: Blackwell. http://www.digitalhumanities.org/companionDLS/

This article argues for and sketches out the basic principles of quantitative stylometrics, but also includes examples of statistical analysis. (The book also contains chapters on many other aspects of digital literary studies, including a bibliography of online tools and archives.)

http://www.stats.ox.ac.uk/~marchini/teaching/

This online archive of Jonathan Marchini’s statistics teaching notes provides introductions to basic statistical measures; probability; specific probability models including the binomial, Poisson, and normal distributions; hypothesis testing (including chi-squared tests); error probability; and confidence intervals.

Intro to statistics with Sebastian Thrun: https://www.class-central.com/mooc/361/udacity-intro-to-statistics

Statistics: The science of decisions, with Sean Laraway, Ronald Rogers, and Katie Kormanik https://www.class-central.com/mooc/631/udacity-statistics-the-science-of-decisions

Intro to descriptive statistics, with Sean Laraway, Ronald Rogers, and Katie Kormanik https://www.class-central.com/mooc/2309/udacity-intro-to-descriptive-statistics

Intro to inferential statistics (follows on from descriptive stats), with Sean Laraway, Ronald Rogers, and Katie Kormanik https://www.class-central.com/mooc/2310/udacity-intro-to-inferential-statistics#reviews

Stanford’s four statistics MOOCs are a good way to familiarise yourself with the basics of describing and drawing conclusions from data. They also offer an intro to data science, and courses on data analysis with the open-source package R and programming foundations with Python. See https://www.class-central.com/provider/udacity for the full list.