Get Starting with text pre-processing

import pandas as pd import numpy as np import matplotlib.pyplot as plt %matplotlib inline import seaborn as sns print('Setup completed!!')

Setup completed!!

data = pd.read_csv('/work/Womens Clothing E-Commerce Reviews.csv') data.head(10)

Unnamed: 0int64

0 - 9

Clothing IDint64

767 - 1080

767

1080

1077

1049

847

1080

858

1077

data.dtypes

data.drop(['Unnamed: 0','Clothing ID'], axis=1, inplace=True)

data.rename(columns={'Review Text': 'text', 'Positive Feedback Count': 'feedback_count', 'Division Name': 'Division', 'Department Name': 'Department', 'Class Name':'Class', 'Recommended IND': 'Recommended'},inplace=True)

print(data.columns)

Index(['Age', 'Title', 'text', 'Rating', 'Recommended', 'feedback_count',
       'Division', 'Department', 'Class'],
      dtype='object')

data.isnull().sum()

data.dropna(axis=0, inplace=True)

data.isnull().sum()

sns.histplot(data=data, x='Age', kde=True) plt.show()

sns.histplot(data=data, x='Rating', kde=False)

sns.set_style('white') sns.barplot(x=data['Rating'], y=data['Class'],palette="Blues_d")

sns.set_theme(style="white") sns.color_palette("flare", as_cmap=False) sns.barplot(x=data['Age'], y=data['Class'],palette="Reds_d")

sns.displot(data=data, x='Rating', hue="Class", kind="kde", height=6, multiple="fill", clip=(0, None), )

/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:316: UserWarning: Dataset has 0 variance; skipping density estimate. Pass `warn_singular=False` to disable this warning.
  warnings.warn(msg, UserWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:316: UserWarning: Dataset has 0 variance; skipping density estimate. Pass `warn_singular=False` to disable this warning.
  warnings.warn(msg, UserWarning)

sns.boxplot(x=data['Department'], y=data['Age']) plt.show()

data.groupby(['Department']).sum().plot(kind='pie', subplots=False, shadow = False,startangle=90,figsize=(15,10), y='Rating')

sns.countplot(data=data, y='Class', hue='Recommended', palette = "Set1")

sns.countplot(data=data, y='Department', hue='Recommended' ,palette = "Set1")

sns.countplot(data=data, y='Division', hue='Recommended', palette = "Set1")

import nltk from nltk.corpus import stopwords from nltk.stem.porter import PorterStemmer

data.columns

data['all_text'] = data['text']

nltk.download('punkt');

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.

def tokenize(column): # Tokenize a Pandas dataframe columns and returns a list of tokens. tokens = nltk.word_tokenize(column) return [w for w in tokens if w.isalpha]

data['tokenized'] = data.apply(lambda x: tokenize(x['all_text']), axis=1) data[['text', 'tokenized']].head()

textobject

tokenizedobject

I had such high hopes for this dress and really wanted it to work for me. i initially ordered the p…

['I', 'had', 'such', 'high', 'hopes', 'for', 'this', 'dress', 'and', 'really', 'wanted', 'it', 'to'…

I love, love, love this jumpsuit. it's fun, flirty, and fabulous! every time i wear it, i get nothi…

['I', 'love', ',', 'love', ',', 'love', 'this', 'jumpsuit', '.', 'it', "'s", 'fun', ',', 'flirty', …

This shirt is very flattering to all due to the adjustable front tie. it is the perfect length to w…

['This', 'shirt', 'is', 'very', 'flattering', 'to', 'all', 'due', 'to', 'the', 'adjustable', 'front…

I love tracy reese dresses, but this one is not for the very petite. i am just under 5 feet tall an…

['I', 'love', 'tracy', 'reese', 'dresses', ',', 'but', 'this', 'one', 'is', 'not', 'for', 'the', 'v…

I aded this in my basket at hte last mintue to see what it would look like in person. (store pick u…

['I', 'aded', 'this', 'in', 'my', 'basket', 'at', 'hte', 'last', 'mintue', 'to', 'see', 'what', 'it…

nltk.download('stopwords');

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.

def remove_stopwords(tokenized_column): # this fuction will retun a list of tokens with English stopwords removed stops = set(stopwords.words('english')) return [word for word in tokenized_column if not word in stops]

data['stopwords_removed'] = data.apply(lambda x: remove_stopwords(x['tokenized']), axis=1) data[['text', 'stopwords_removed']].head()

textobject

stopwords_removedobject

I had such high hopes for this dress and really wanted it to work for me. i initially ordered the p…

['I', 'high', 'hopes', 'dress', 'really', 'wanted', 'work', '.', 'initially', 'ordered', 'petite', …

I love, love, love this jumpsuit. it's fun, flirty, and fabulous! every time i wear it, i get nothi…

['I', 'love', ',', 'love', ',', 'love', 'jumpsuit', '.', "'s", 'fun', ',', 'flirty', ',', 'fabulous…

This shirt is very flattering to all due to the adjustable front tie. it is the perfect length to w…

['This', 'shirt', 'flattering', 'due', 'adjustable', 'front', 'tie', '.', 'perfect', 'length', 'wea…

I love tracy reese dresses, but this one is not for the very petite. i am just under 5 feet tall an…

['I', 'love', 'tracy', 'reese', 'dresses', ',', 'one', 'petite', '.', '5', 'feet', 'tall', 'usually…

I aded this in my basket at hte last mintue to see what it would look like in person. (store pick u…

['I', 'aded', 'basket', 'hte', 'last', 'mintue', 'see', 'would', 'look', 'like', 'person', '.', '('…

def apply_stemming(tokenized_column): # this function will return a list of tokens with PorterStemming applied stemmer = PorterStemmer() return [stemmer.stem(word) for word in tokenized_column]

data['porter_stemmed'] = data.apply(lambda x: apply_stemming(x['stopwords_removed']), axis=1) data[['text', 'porter_stemmed']].head()

textobject

porter_stemmedobject

I had such high hopes for this dress and really wanted it to work for me. i initially ordered the p…

['i', 'high', 'hope', 'dress', 'realli', 'want', 'work', '.', 'initi', 'order', 'petit', 'small', '…

I love, love, love this jumpsuit. it's fun, flirty, and fabulous! every time i wear it, i get nothi…

['i', 'love', ',', 'love', ',', 'love', 'jumpsuit', '.', "'s", 'fun', ',', 'flirti', ',', 'fabul', …

This shirt is very flattering to all due to the adjustable front tie. it is the perfect length to w…

['thi', 'shirt', 'flatter', 'due', 'adjust', 'front', 'tie', '.', 'perfect', 'length', 'wear', 'leg…

I love tracy reese dresses, but this one is not for the very petite. i am just under 5 feet tall an…

['i', 'love', 'traci', 'rees', 'dress', ',', 'one', 'petit', '.', '5', 'feet', 'tall', 'usual', 'we…

I aded this in my basket at hte last mintue to see what it would look like in person. (store pick u…

['i', 'ade', 'basket', 'hte', 'last', 'mintu', 'see', 'would', 'look', 'like', 'person', '.', '(', …

def rejoin_words(tokenized_column): # this fuction will rejoin the tokenized words list into a single string return (" ".join(tokenized_column))

data['rejoined'] = data.apply(lambda x: rejoin_words(x['porter_stemmed']), axis=1) data[['text', 'rejoined']].head()

textobject

rejoinedobject

I had such high hopes for this dress and really wanted it to work for me. i initially ordered the p…

i high hope dress realli want work . initi order petit small ( usual size ) found outrag small . sm…

I love, love, love this jumpsuit. it's fun, flirty, and fabulous! every time i wear it, i get nothi…

i love , love , love jumpsuit . 's fun , flirti , fabul ! everi time wear , get noth great complime…

This shirt is very flattering to all due to the adjustable front tie. it is the perfect length to w…

thi shirt flatter due adjust front tie . perfect length wear leg sleeveless pair well cardigan . lo…

I love tracy reese dresses, but this one is not for the very petite. i am just under 5 feet tall an…

i love traci rees dress , one petit . 5 feet tall usual wear 0p brand . dress pretti packag lot dre…

I aded this in my basket at hte last mintue to see what it would look like in person. (store pick u…

i ade basket hte last mintu see would look like person . ( store pick ) . went teh darkler color pa…