Blueprint: Standardizing Attribute Names

Before we start working with the data, we will change the dataset-specific column names to more generic names. We recommend always naming the main DataFrame df, and naming the column with the text to analyze text. Such naming conventions for common variables and attribute names make it easier to reuse the code of the blueprints in different projects.

Let’s take a look at the columns list of this dataset:

print(df.columns)

Out:

Index(['id', 'subreddit', 'title', 'selftext', 'category_1', 'category_2',
       'category_3', 'in_data', 'reason_for_exclusion'],
      dtype='object')

For column renaming and selection, we define a dictionary column_mapping where each entry defines a mapping from the current column name to a new name. Columns mapped to None and unmentioned columns are dropped. A dictionary is perfect documentation for such a transformation and easy to reuse. This dictionary is then used to select and rename the columns that we want to keep.

column_mapping = {
    'id': 'id',
    'subreddit': 'subreddit',
    'title': 'title',
    'selftext': 'text',
    'category_1': 'category',
    'category_2': 'subcategory',
    'category_3': None, # no data
    'in_data': None, # not needed
    'reason_for_exclusion': None # not needed
}

# define remaining columns
columns = [c for c in column_mapping.keys() if column_mapping[c] != None]

# select and rename those columns
df = df[columns].rename(columns=column_mapping)

As already mentioned, we limit the data to the autos category:

df = df[df['category'] == 'autos']

Let’s take a brief look at a sample record to get a first impression of the data:

df.sample(1).T

	14356
id	7jc2k4
subreddit	volt
title	Dashcam for 2017 volt
text	Hello.<lb>I’m looking into getting a dashcam. <lb>Does anyone have any recommendations? <lb><lb>I’m generally looking for a rechargeable one so that I don’t have to route wires down to the cigarette lighter. <lb>Unless there are instructions on how to wire it properly without wires showing. <lb><lb><lb>Thanks!
category	autos
subcategory	chevrolet

Blueprint: Tokenization with Regular Expressions

Useful functions for tokenization are re.split() and re.findall(). The first one splits a string at matching expressions, while the latter extracts all character sequences matching a certain pattern. For example, in Chapter 1 we used the regex library with the POSIX pattern [\w-]*\p{L}[\w-]* to find sequences of alphanumeric characters with at least one letter. The scikit-learn CountVectorizer uses the pattern \w\w+ for its default tokenization. It matches all sequences of two or more alphanumeric characters. Applied to our sample sentence, this yields the following result:³

tokens = re.findall(r'\w\w+', text)
print(*tokens, sep='|')

Out:

2019|08|10|23|32|pete|louis|don|have|well|designed|solution|for|today
problem|The|code|of|module|AC68|should|be|Have|to|think|bit|goodnight

Unfortunately, all special characters and the emojis are lost. To improve the result, we add some additional expressions for the emojis and create a reusable regular expression RE_TOKEN. The VERBOSE option allows readable formatting of complex expressions. The following tokenize function and the example illustrate the use:

RE_TOKEN = re.compile(r"""
               ( [#]?[@\w'’\.\-\:]*\w     # words, hashtags and email addresses
               | [:;<]\-?[\)\(3]          # coarse pattern for basic text emojis
               | [\U0001F100-\U0001FFFF]  # coarse code range for unicode emojis
               )
               """, re.VERBOSE)

def tokenize(text):
    return RE_TOKEN.findall(text)

tokens = tokenize(text)
print(*tokens, sep='|')

Out:

2019-08-10|23:32|@pete|@louis|I|don't|have|a|well-designed|solution
for|today's|problem|The|code|of|module|AC68|should|be|-1|Have|to|think
a|bit|#goodnight|;-)|😩|😬

This expression should yield reasonably good results on most user-generated content. It can be used to quickly tokenize text for data exploration, as explained in Chapter 1. It’s also a good alternative for the default tokenization of the scikit-learn vectorizers, which will be introduced in the next chapter.

Blueprint: Customizing Tokenization

Tokenization is the first step in the pipeline, and everything depends on the correct tokens. spaCy’s tokenizer does a good job in most cases, but it splits on hash signs, hyphens, and underscores, which is sometimes not what you want. Therefore, it may be necessary to adjust its behavior. Let’s look at the following text as an example:

text = "@Pete: choose low-carb #food #eat-smart. _url_ ;-) 😋👍"
doc = nlp(text)

for token in doc:
    print(token, end="|")

Out:

@Pete|:|choose|low|-|carb|#|food|#|eat|-|smart|.|_|url|_|;-)|😋|👍|

spaCy’s tokenizer is completely rule-based. First, it splits the text on whitespace characters. Then it uses prefix, suffix, and infix splitting rules defined by regular expressions to further split the remaining tokens. Exception rules are used to handle language-specific exceptions like can’t, which should be split into ca and n’t with lemmas can and not.⁹

As you can see in the example, spaCy’s English tokenizer contains an infix rule for splits at hyphens. In addition, it has a prefix rule to split off characters like # or _. It works well for tokens prefixed with @ and emojis, though.

One option is to merge tokens in a postprocessing step using doc.retokenize. However, that will not fix any miscalculated part-of-speech tags and syntactical dependencies because these rely on tokenization. So it may be better to change the tokenization rules and create correct tokens in the first place.

The best approach for this is to create your own variant of the tokenizer with individual rules for infix, prefix, and suffix splitting.¹⁰ The following function creates a tokenizer object with individual rules in a “minimally invasive” way: we just drop the respective patterns from spaCy’s default rules but retain the major part of the logic:

from spacy.tokenizer import Tokenizer
from spacy.util import compile_prefix_regex, \
                       compile_infix_regex, compile_suffix_regex

def custom_tokenizer(nlp):

    # use default patterns except the ones matched by re.search
    prefixes = [pattern for pattern in nlp.Defaults.prefixes
                if pattern not in ['-', '_', '#']]
    suffixes = [pattern for pattern in nlp.Defaults.suffixes
                if pattern not in ['_']]
    infixes  = [pattern for pattern in nlp.Defaults.infixes
                if not re.search(pattern, 'xx-xx')]

    return Tokenizer(vocab          = nlp.vocab,
                     rules          = nlp.Defaults.tokenizer_exceptions,
                     prefix_search  = compile_prefix_regex(prefixes).search,
                     suffix_search  = compile_suffix_regex(suffixes).search,
                     infix_finditer = compile_infix_regex(infixes).finditer,
                     token_match    = nlp.Defaults.token_match)

nlp = spacy.load('en_core_web_sm')
nlp.tokenizer = custom_tokenizer(nlp)

doc = nlp(text)
for token in doc:
    print(token, end="|")

Out:

@Pete|:|choose|low-carb|#food|#eat-smart|.|_url_|;-)|😋|👍|

Blueprint: Working with Stop Words

spaCy uses language-specific stop word lists to set the is_stop property for each token directly after tokenization. Thus, filtering stop words (and similarly punctuation tokens) is easy:

text = "Dear Ryan, we need to sit down and talk. Regards, Pete"
doc = nlp(text)

non_stop = [t for t in doc if not t.is_stop and not t.is_punct]
print(non_stop)

Out:

[Dear, Ryan, need, sit, talk, Regards, Pete]

The list of English stop words with more than 300 entries can be accessed by importing spacy.lang.en.STOP_WORDS. When an nlp object is created, this list is loaded and stored under nlp.Defaults.stop_words. We can modify spaCy’s default behavior by setting the is_stop property of the respective words in spaCy’s vocabulary:¹¹

nlp = spacy.load('en_core_web_sm')
nlp.vocab['down'].is_stop = False
nlp.vocab['Dear'].is_stop = True
nlp.vocab['Regards'].is_stop = True

If we rerun the previous example, we get the following result:

[Ryan, need, sit, down, talk, Pete]

Blueprint: Extracting Lemmas Based on Part of Speech

Lemmatization is the mapping of a word to its uninflected root. Treating words like housing, housed, and house as the same has many advantages for statistics, machine learning, and information retrieval. It can not only improve the quality of the models but also decrease training time and model size because the vocabulary is much smaller if only uninflected forms are kept. In addition, it is often helpful to restrict the types of the words used to certain categories, such as nouns, verbs, and adjectives. Those word types are called part-of-speech tags.

Let’s first take a closer look at lemmatization. The lemma of a token or span can be accessed by the lemma_ property, as illustrated in the following example:

text = "My best friend Ryan Peters likes fancy adventure games."
doc = nlp(text)

print(*[t.lemma_ for t in doc], sep='|')

Out:

-PRON-|good|friend|Ryan|Peters|like|fancy|adventure|game|.

The correct assignment of the lemma requires a lookup dictionary and knowledge about the part of speech of a word. For example, the lemma of the noun meeting is meeting, while the lemma of the verb is meet. In English, spaCy is able to make this distinction. In most other languages, however, lemmatization is purely dictionary-based, ignoring the part-of-speech dependency. Note that personal pronouns like I, me, you, and her always get the lemma -PRON- in spaCy.

The other token attribute we will use in this blueprint is the part-of-speech tag. Table 4-3 shows that each token in a spaCy doc has two part-of-speech attributes: pos_ and tag_. tag_ is the tag from the tagset used to train the model. For spaCy’s English models, which have been trained on the OntoNotes 5 corpus, this is the Penn Treebank tagset. For a German model, this would be the Stuttgart-Tübingen tagset. The pos_ attribute contains the simplified tag of the universal part-of-speech tagset.¹² We recommend using this attribute as the values will remain stable across different models. Table 4-4 shows the complete tag set descriptions.

Table 4-4. Universal part-of-speech tags

Tag	Description	Examples
ADJ	Adjectives (describe nouns)	big, green, African
ADP	Adpositions (prepositions and postpositions)	in, on
ADV	Adverbs (modify verbs or adjectives)	very, exactly, always
AUX	Auxiliary (accompanies verb)	can (do), is (doing)
CCONJ	Connecting conjunction	and, or, but
DET	Determiner (with regard to nouns)	the, a, all (things), your (idea)
INTJ	Interjection (independent word, exclamation, expression of emotion)	hi, yeah
NOUN	Nouns (common and proper)	house, computer
NUM	Cardinal numbers	nine, 9, IX
PROPN	Proper noun, name, or part of a name	Peter, Berlin
PRON	Pronoun, substitute for noun	I, you, myself, who
PART	Particle (makes sense only with other word)
PUNCT	Punctuation characters	, . ;
SCONJ	Subordinating conjunction	before, since, if
SYM	Symbols (word-like)	$, ©
VERB	Verbs (all tenses and modes)	go, went, thinking
X	Anything that cannot be assigned	grlmpf

Part-of-speech tags are an excellent alternative to stop words as word filters. In linguistics, pronouns, prepositions, conjunctions, and determiners are called function words because their main function is to create grammatical relationships within a sentence. Nouns, verbs, adjectives, and adverbs are content words, and the meaning of a sentence depends mainly on them.

Often, we are interested only in content words. Thus, instead of using a stop word list, we can use part-of-speech tags to select the word types we are interested in and discard the rest. For example, a list containing only the nouns and proper nouns in a doc can be generated like this:

text = "My best friend Ryan Peters likes fancy adventure games."
doc = nlp(text)

nouns = [t for t in doc if t.pos_ in ['NOUN', 'PROPN']]
print(nouns)

Out:

[friend, Ryan, Peters, adventure, games]

We could easily define a more general filter function for this purpose, but textacy’s extract.words function conveniently provides this functionality. It also allows us to filter on part of speech and additional token properties such as is_punct or is_stop. Thus, the filter function allows both part-of-speech selection and stop word filtering. Internally it works just like we illustrated for the noun filter shown previously.

The following example shows how to extract tokens for adjectives and nouns from the sample sentence:

import textacy

tokens = textacy.extract.words(doc,
            filter_stops = True,           # default True, no stopwords
            filter_punct = True,           # default True, no punctuation
            filter_nums = True,            # default False, no numbers
            include_pos = ['ADJ', 'NOUN'], # default None = include all
            exclude_pos = None,            # default None = exclude none
            min_freq = 1)                  # minimum frequency of words

print(*[t for t in tokens], sep='|')

Out:

best|friend|fancy|adventure|games

Our blueprint function to extract a filtered list of word lemmas is finally just a tiny wrapper around that function. By forwarding the keyword arguments (**kwargs), this function accepts the same parameters as textacy’s extract.words.

def extract_lemmas(doc, **kwargs):
    return [t.lemma_ for t in textacy.extract.words(doc, **kwargs)]

lemmas = extract_lemmas(doc, include_pos=['ADJ', 'NOUN'])
print(*lemmas, sep='|')

Out:

good|friend|fancy|adventure|game

Blueprint: Extracting Named Entities

Named-entity recognition refers to the process of detecting entities such as people, locations, or organizations in text. Each entity can consist of one or more tokens, like San Francisco. Therefore, named entities are represented by Span objects. As with noun phrases, it can be helpful to retrieve a list of named entities for further analysis.

If you look again at Table 4-3, you see the token attributes for named-entity recognition, ent_type_ and ent_iob_. ent_iob_ contains the information if a token begins an entity (B), is inside an entity (I), or is outside (O). Instead of iterating through the tokens, we can also access the named entities directly with doc.ents. Here, the property for the entity type is called label_. Let’s illustrate this with an example:

text = "James O'Neill, chairman of World Cargo Inc, lives in San Francisco."
doc = nlp(text)

for ent in doc.ents:
    print(f"({ent.text}, {ent.label_})", end=" ")

Out:

(James O'Neill, PERSON) (World Cargo Inc, ORG) (San Francisco, GPE)

spaCy’s displacy module also provides visualization for the named-entity recognition, which makes the result much more readable and visually supports the identification of misclassified entities:

from spacy import displacy

displacy.render(doc, style='ent')

The named entities were identified correctly as a person, an organization, and a geo-political entity (GPE). But be aware that the accuracy for named-entity recognition may not be very good if your corpus is missing a clear grammatical structure. Check out “Named-Entity Recognition” for a detailed discussion.

For the extraction of named entities of certain types, we again make use of one of textacy’s convenient functions:

def extract_entities(doc, include_types=None, sep='_'):

    ents = textacy.extract.entities(doc,
             include_types=include_types,
             exclude_types=None,
             drop_determiners=True,
             min_freq=1)

    return [sep.join([t.lemma_ for t in e])+'/'+e.label_ for e in ents]

With this function we can, for example, retrieve the named entities of types PERSON and GPE (geo-political entity) like this:

print(extract_entities(doc, ['PERSON', 'GPE']))

Out:

["James_O'Neill/PERSON", 'San_Francisco/GPE']

Blueprint: Using spaCy on a Large Dataset

Now we can use this function to extract features from all the records of a dataset. We take the cleaned texts that we created and saved at the beginning of this chapter and add the titles:

db_name = "reddit-selfposts.db"
con = sqlite3.connect(db_name)
df = pd.read_sql("select * from posts_cleaned", con)
con.close()

df['text'] = df['title'] + ': ' + df['text']

Before we start NLP processing, we initialize the new DataFrame columns we want to fill with values:

for col in nlp_columns:
    df[col] = None

spaCy’s neural models benefit from running on GPU. Thus, we try to load the model on the GPU before we start:

if spacy.prefer_gpu():
    print("Working on GPU.")
else:
    print("No GPU found, working on CPU.")

Now we have to decide which model and which of the pipeline components to use. Remember to disable unneccesary components to improve runtime! We stick to the small English model with the default pipeline and use our custom tokenizer that splits on hyphens:

nlp = spacy.load('en_core_web_sm', disable=[])
nlp.tokenizer = custom_tokenizer(nlp) # optional

When processing larger datasets, it is recommended to use spaCy’s batch processing for a significant performance gain (roughly factor 2 on our dataset). The function nlp.pipe takes an iterable of texts, processes them internally as a batch, and yields a list of processed Doc objects in the same order as the input data.

To use it, we first have to define a batch size. Then we can loop over the batches and call nlp.pipe.

batch_size = 50

for i in range(0, len(df), batch_size):
    docs = nlp.pipe(df['text'][i:i+batch_size])

    for j, doc in enumerate(docs):
        for col, values in extract_nlp(doc).items():
            df[col].iloc[i+j] = values

In the inner loop we extract the features from the processed doc and write the values back into the DataFrame. The whole process takes about six to eight minutes on the dataset without using a GPU and about three to four minutes with the GPU on Colab.

The newly created columns are perfectly suited for frequency analysis with the functions from Chapter 1. Let’s check for the most frequently mentioned noun phrases in the autos category:

count_words(df, 'noun_phrases').head(10).plot(kind='barh').invert_yaxis()

Out: