Amazon Reviews Sentiment Analysis

Import the Libraries

import requests from bs4 import BeautifulSoup import pandas as pd from urllib.parse import urlencode import csv

requests allows you to send HTTP requests to a server that returns a Response Object with all the response data (i.e. HTML). BeautifulSoup (bs4) is used to pull data out of HTML files and convert the data to a BeautifulSoup object, which represents the HTML as a nested data structure. pandas is used for data analysis and manipulation. urllib can be used for many purposes, including reading website content, making HTTP and HTTPS requests, sending request headers, and retrieving response headers. csv module implements classes to read and write tabular data in CSV format.

Review the Web Page’s HTML Structure

We need to understand the structure and contents of the HTML tags within the web pages. We will be using the Amazon webpage and searching for Apple Watch Series 7 GPS + Cellular (shown below). You can find this webpage by selecting this link and scrolling down to the reviews section.

We can scrape this webpage by parsing the HTML of the page and extracting the information needed for our dataset

we will extract the following data elements from the reviews:

Reviewer Names Reviews

Retrieve HTML data and Extract the Data Elements

# Define a list of URL's that will be scraped. list_of_urls = ['https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_othr_d_show_all_btm?ie=UTF8&reviewerType=all_reviews', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=2', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_getr_d_paging_btm_next_3?ie=UTF8&reviewerType=all_reviews&pageNumber=3', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_getr_d_paging_btm_next_4?ie=UTF8&reviewerType=all_reviews&pageNumber=4', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_getr_d_paging_btm_next_5?ie=UTF8&reviewerType=all_reviews&pageNumber=5']

# Retrieve each of the url's HTML data and convert the data into a beautiful soup object. # Find, extract and store reviewer names and review text into a list. names = [] reviews = [] data_string = "" for url in list_of_urls: params = {'api_key': "ce7a385c717c6948e16f7ccb5b427d9e", 'url': url} response = requests.get('http://api.scraperapi.com/', params=urlencode(params)) soup = BeautifulSoup(response.text, 'html.parser') for item in soup.find_all("span", class_="a-profile-name"): data_string = data_string + item.get_text() names.append(data_string) data_string = "" for item in soup.find_all("span", {"data-hook": "review-body"}): data_string = data_string + item.get_text() reviews.append(data_string) data_string = ""

Create the Dictionary

We will create a dictionary that will contain the data names and values for the data elements that were extracted.

# Create the dictionary. reviews_dict = {'Reviewer Name': names, 'Reviews': reviews} # Print the lengths of each list. print(len(names), len(reviews))

54 44

Create the Data Frame

We need to adjust for missing values in columns. This step will create the data frame as each key was a row and like this, the missing values are actually missing columns which is no problem for pandas (only missing rows lead to ValueError during creation). We need to transpose the data frame (flip the axis) and change the rows to columns. We also need to clean some of the data.

# Create a new dataframe. df = pd.DataFrame.from_dict(reviews_dict, orient='index') df.head()

0object

1object

Reviewer Name

David

Makenna Meyers

Reviews

I can’t tell this was a refurb. The packaging was new, looked like it had never been touched. Very pleased.

Good product and very nice size

# Delete all the columns that have missing values. df.dropna(axis=1, inplace=True) df.head()

0object

1object

Reviewer Name

David

Makenna Meyers

Reviews

I can’t tell this was a refurb. The packaging was new, looked like it had never been touched. Very pleased.

Good product and very nice size

# Transpose the dataframe. prod_reviews = df.T print(prod_reviews.head(10))

       Reviewer Name                                            Reviews
0              David  \nI can’t tell this was a refurb. The packagin...
1     Makenna Meyers                \nGood product and very nice size\n
2              David  \nEverything looks and feels new for a resale ...
3          Florencia  \nGreat product. Syncs well with my iPhone. Ba...
4    Amazon Customer  \nBought this for myself months ago, watch cam...
5     Anthony Taylor                                    \nI love it .\n
6  Austin Richardson  \nOrdered this refurbished watch and the watch...
7         Denisnunez  \nI can’t even charge my watch right off the b...
8        Amanda Boyd                  \nGift to my niece who is an RN\n
9      Sheila Larson                                    \nWorks great\n

# Remove special characters from review text. prod_reviews['Reviews'] = prod_reviews['Reviews'].str.replace('\n','') prod_reviews.head(5)

Reviewer Nameobject

Reviewsobject

David

I can’t tell this was a refurb. The packaging was new, looked like it had never been touched. Very pleased.

Makenna Meyers

Good product and very nice size

David

Everything looks and feels new for a resale item

Florencia

Great product. Syncs well with my iPhone. Battery life is marginal.

Amazon Customer

Bought this for myself months ago, watch came in the original apple packaging with a usb-c charger, amazing quality for the money. decided to purchase again for my girlfriend a couple months later. this time the watch didn’t come in the original apple watch packaging and had a traditional usb charger, still came with the authentic watch band and watch itself, would still say it was very much worth the money and a great buy.

prod_reviews.shape

# Convert dataframe to CSV file. prod_reviews.to_csv('dataset/reviews.csv', index=False, header=True)

Sentiment Analysis

Sentiment Analysis is a use case of Natural Language Processing (NLP) and comes under the category of text classification. To put it simply, Sentiment Analysis involves classifying a text into various sentiments, such as positive or negative, Happy, Sad or Neutral, etc. Thus, the ultimate goal of sentiment analysis is to decipher the underlying mood, emotion, or sentiment of a text. This is also known as Opinion Mining

#pip install transformers

#import transformers

from textblob import TextBlob

data=TextBlob(prod_reviews.Reviews[5]) print(data.sentiment) print(prod_reviews.Reviews[5]) if 0<data.sentiment[0]<0.5 : print('Neutral')

Sentiment(polarity=0.5, subjectivity=0.6)
I love it .

We can see the output is categorized between two — Polarity and Subjectivity.

Polarity is a float value within the range [-1.0 to 1.0] where 0 indicates neutral, +1 indicates a very positive sentiment and -1 represents a very negative sentiment.

Subjectivity is a float value within the range [0.0 to 1.0] where 0.0 is very objective and 1.0 is very subjective. Subjective sentence expresses some personal feelings, views, beliefs, opinions, allegations, desires, beliefs, suspicions, and speculations where as Objective sentences are factual.

prod_reviews['Review Type']=0 for i in range(len(prod_reviews)): data=TextBlob(prod_reviews.Reviews[i]) if data.sentiment[0]>=0.5: prod_reviews['Review Type'][i]='Positive' if 0<=data.sentiment[0]<0.5 : prod_reviews['Review Type'][i]='Neutral' if data.sentiment[0]<0 : prod_reviews['Review Type'][i]='Negative'

/tmp/ipykernel_124/512394660.py:7: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  prod_reviews['Review Type'][i]='Neutral'
/shared-libs/python3.9/py/lib/python3.9/site-packages/pandas/core/indexing.py:1637: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_single_block(indexer, value, name)

prod_reviews

Reviewer Nameobject

David11.4%

Makenna Meyers11.4%

32 others77.3%

Reviewsobject

4.5%

I can’t tell this was a refurb. The packaging was new, looked like it had never been touched. Very pleased.2.3%

41 others93.2%

David

I can’t tell this was a refurb. The packaging was new, looked like it had never been touched. Very pleased.

Makenna Meyers

Good product and very nice size

David

Everything looks and feels new for a resale item

Florencia

Great product. Syncs well with my iPhone. Battery life is marginal.

Amazon Customer

Anthony Taylor

I love it .

Austin Richardson

Ordered this refurbished watch and the watch is great! The main problem I had with it was that it came with an aftermarket charger. This charger WOULD NOT charge the watch. It would start charging, then stop after a few seconds, then start again, and would keep doing this until I pulled it off the charger. I ordered an actual apple charger for the watch and the watch was charged in a matter of an hour or so. I highly suggest ordering an apple charger at the same time you order the refurbished watch.

Denisnunez

I can’t even charge my watch right off the bat. Thoroughly disappointed that they weren’t able to have quality control over this. Now I have to drive to town to get the correct charging block.

Amanda Boyd

Gift to my niece who is an RN

Sheila Larson

Works great

res=pd.DataFrame(prod_reviews['Review Type'].value_counts()) res.plot.pie(y='Review Type',autopct='%1.1f%%',title="Review Types")

.css-ytumd6{-webkit-text-decoration:none;text-decoration:none;}Amazon Reviews Sentiment Analysis

Import the Libraries

Review the Web Page’s HTML Structure

Retrieve HTML data and Extract the Data Elements

Create the Dictionary

Create the Data Frame

Sentiment Analysis

Amazon Reviews Sentiment Analysis