HW01: Intro to Text Data

%%capture !pip install plox !pip install tabulate

!date

In this assignment, we will explore how to load a text classification dataset (AG's news, originally posted here), how we can preprocess the data and extract useful information from a real-world dataset. First, we have to download the data; we only download a subset of the data with four classes.

!wget https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/train.csv

Inspect Data

import pandas as pd df = pd.read_csv("train.csv", header=None) df.info() df.head()

Let's make the data more human readable by adding a header and replacing labels

df.columns = ["label", "title", "lead"] label_map = {1: "world", 2: "sport", 3: "business", 4: "sci/tech"} df["label"] = df["label"].apply(label_map.get)

df.head()

# a new column `text` which contains the lowercased `title` and `lead` df["text"] = (df.title + " " + df.lead).str.lower() print(df.text[0])

# print the number of documents for each label df.label.groupby(df.label).count()

Document Length

# create a new column with the number of words for each text # plot the average number of words per label import re from plox import Plox def words(text: str): return list(filter(len, re.findall("([a-z][a-z'-]*)", str.lower(text)))) df['nwords'] = df.text.apply(lambda text: len(words(text))) avg_nwords = df.groupby(by='label').nwords.mean() with Plox() as px: px.a.bar(x=avg_nwords.index, height=avg_nwords) px.a.set_ylabel("Average number of words") px.a.set_xlabel("Text label") px.show()

Word Frequency

Let's implement a keyword search (similar to the baker-bloom economic uncertainty) and compute how often some given keywords ("play", "tax", "blackberry", "israel") appear in the different classes in our data

from collections import Counter keywords = ["play", "tax", "blackberry", "israel"] counts = pd.DataFrame( data={ keyword: df.text.apply( lambda text: Counter(words(text))[keyword] ).groupby(df.label).sum() for keyword in keywords }, ) counts['total'] = counts.T.sum(axis=1) print(counts.to_markdown())

As a last exercise, let's plot the number of occurrences of "tax" in the different classes in the dataset. Hint: have a look at the pandas bar plot with group by

with Plox() as px: counts.plot(kind='bar', y='tax', ax=px.a) px.show()

HW01: Intro to Text Data

Inspect Data

Document Length

Word Frequency

Recommended on Deepnote

Stock Market Analysis

The 10 Best Ways to Create NumPy Arrays

Wide Residual Networks