%%capture
!pip install plox
!pip install tabulate
!date
Mon 22 Feb 2021 07:49:20 PM UTC
!wget https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/train.csv
--2021-02-22 18:49:43-- https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/train.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29470338 (28M) [text/plain]
Saving to: ‘train.csv’
train.csv 100%[===================>] 28.10M --.-KB/s in 0.1s
2021-02-22 18:49:44 (269 MB/s) - ‘train.csv’ saved [29470338/29470338]
import pandas as pd
df = pd.read_csv("train.csv", header=None)
df.info()
df.head()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 120000 entries, 0 to 119999
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 0 120000 non-null int64
1 1 120000 non-null object
2 2 120000 non-null object
dtypes: int64(1), object(2)
memory usage: 2.7+ MB
| 0int64 | 1object | 2object | ||
|---|---|---|---|---|
0 | 3 | Wall St. Bears Claw Back Into the Black (Reuters) | Reuters - Short-sellers, Wall Street's dwindling\band of ultra-cynics, are seeing green again. | |
1 | 3 | Carlyle Looks Toward Commercial Aerospace (Reuters) | Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market. | |
2 | 3 | Oil and Economy Cloud Stocks' Outlook (Reuters) | Reuters - Soaring crude prices plus worries\about the economy and the outlook for earnings are expected to\hang over the stock market next week during the depth of the\summer doldrums. | |
3 | 3 | Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) | Reuters - Authorities have halted oil export\flows from the main pipeline in southern Iraq after\intelligence showed a rebel militia could strike\infrastructure, an oil official said on Saturday. | |
4 | 3 | Oil prices soar to all-time record, posing new menace to US economy (AFP) | AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections. | |
| 5 rows × 3 columns | ||||
df.columns = ["label", "title", "lead"]
label_map = {1: "world", 2: "sport", 3: "business", 4: "sci/tech"}
df["label"] = df["label"].apply(label_map.get)
df.head()
| labelobject | titleobject | leadobject | textobject | ||
|---|---|---|---|---|---|
0 | business | Wall St. Bears Claw Back Into the Black (Reuters) | Reuters - Short-sellers, Wall Street's dwindling\band of ultra-cynics, are seeing green again. | Wall St. Bears Claw Back Into the Black (Reuters)Reuters - Short-sellers, Wall Street's dwindling\band of ultra-cynics, are seeing green again. | |
1 | business | Carlyle Looks Toward Commercial Aerospace (Reuters) | Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market. | Carlyle Looks Toward Commercial Aerospace (Reuters)Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market. | |
2 | business | Oil and Economy Cloud Stocks' Outlook (Reuters) | Reuters - Soaring crude prices plus worries\about the economy and the outlook for earnings are expected to\hang over the stock market next week during the depth of the\summer doldrums. | Oil and Economy Cloud Stocks' Outlook (Reuters)Reuters - Soaring crude prices plus worries\about the economy and the outlook for earnings are expected to\hang over the stock market next week during the depth of the\summer doldrums. | |
3 | business | Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) | Reuters - Authorities have halted oil export\flows from the main pipeline in southern Iraq after\intelligence showed a rebel militia could strike\infrastructure, an oil official said on Saturday. | Iraq Halts Oil Exports from Main Southern Pipeline (Reuters)Reuters - Authorities have halted oil export\flows from the main pipeline in southern Iraq after\intelligence showed a rebel militia could strike\infrastructure, an oil official said on Saturday. | |
4 | business | Oil prices soar to all-time record, posing new menace to US economy (AFP) | AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections. | Oil prices soar to all-time record, posing new menace to US economy (AFP)AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections. | |
| 5 rows × 4 columns | |||||
# a new column `text` which contains the lowercased `title` and `lead`
df["text"] = (df.title + " " + df.lead).str.lower()
print(df.text[0])
wall st. bears claw back into the black (reuters) reuters - short-sellers, wall street's dwindling\band of ultra-cynics, are seeing green again.
# print the number of documents for each label
df.label.groupby(df.label).count()
# create a new column with the number of words for each text
# plot the average number of words per label
import re
from plox import Plox
def words(text: str):
return list(filter(len, re.findall("([a-z][a-z'-]*)", str.lower(text))))
df['nwords'] = df.text.apply(lambda text: len(words(text)))
avg_nwords = df.groupby(by='label').nwords.mean()
with Plox() as px:
px.a.bar(x=avg_nwords.index, height=avg_nwords)
px.a.set_ylabel("Average number of words")
px.a.set_xlabel("Text label")
px.show()
from collections import Counter
keywords = ["play", "tax", "blackberry", "israel"]
counts = pd.DataFrame(
data={
keyword:
df.text.apply(
lambda text: Counter(words(text))[keyword]
).groupby(df.label).sum()
for keyword in keywords
},
)
counts['total'] = counts.T.sum(axis=1)
print(counts.to_markdown())
| label | play | tax | blackberry | israel | total |
|:---------|-------:|------:|-------------:|---------:|--------:|
| business | 72 | 834 | 43 | 15 | nan |
| sci/tech | 212 | 49 | 181 | 24 | nan |
| sport | 1354 | 9 | 0 | 32 | nan |
| world | 77 | 144 | 0 | 1243 | nan |
with Plox() as px:
counts.plot(kind='bar', y='tax', ax=px.a)
px.show()