Amazon Reviews Sentiment Analysis

Import the Libraries

import requests from bs4 import BeautifulSoup import pandas as pd from urllib.parse import urlencode import csv

requests allows you to send HTTP requests to a server that returns a Response Object with all the response data (i.e. HTML). BeautifulSoup (bs4) is used to pull data out of HTML files and convert the data to a BeautifulSoup object, which represents the HTML as a nested data structure. pandas is used for data analysis and manipulation. urllib can be used for many purposes, including reading website content, making HTTP and HTTPS requests, sending request headers, and retrieving response headers. csv module implements classes to read and write tabular data in CSV format.

Review the Web Page’s HTML Structure

We need to understand the structure and contents of the HTML tags within the web pages. We will be using the Amazon webpage and searching for Apple Watch Series 7 GPS + Cellular (shown below). You can find this webpage by selecting this link and scrolling down to the reviews section.

We can scrape this webpage by parsing the HTML of the page and extracting the information needed for our dataset

we will extract the following data elements from the reviews:

Reviewer Names Reviews

Retrieve HTML data and Extract the Data Elements

# Define a list of URL's that will be scraped. list_of_urls = ['https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_othr_d_show_all_btm?ie=UTF8&reviewerType=all_reviews', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber=2', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_getr_d_paging_btm_next_3?ie=UTF8&reviewerType=all_reviews&pageNumber=3', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_getr_d_paging_btm_next_4?ie=UTF8&reviewerType=all_reviews&pageNumber=4', 'https://www.amazon.com/Apple-Watch-Cellular-Midnight-Aluminum/product-reviews/B09PB1PTH8/ref=cm_cr_getr_d_paging_btm_next_5?ie=UTF8&reviewerType=all_reviews&pageNumber=5']

# Retrieve each of the url's HTML data and convert the data into a beautiful soup object. # Find, extract and store reviewer names and review text into a list. names = [] reviews = [] data_string = "" for url in list_of_urls: params = {'api_key': "ce7a385c717c6948e16f7ccb5b427d9e", 'url': url} response = requests.get('http://api.scraperapi.com/', params=urlencode(params)) soup = BeautifulSoup(response.text, 'html.parser') for item in soup.find_all("span", class_="a-profile-name"): data_string = data_string + item.get_text() names.append(data_string) data_string = "" for item in soup.find_all("span", {"data-hook": "review-body"}): data_string = data_string + item.get_text() reviews.append(data_string) data_string = ""

Create the Dictionary

We will create a dictionary that will contain the data names and values for the data elements that were extracted.

# Create the dictionary. reviews_dict = {'Reviewer Name': names, 'Reviews': reviews} # Print the lengths of each list. print(len(names), len(reviews))

Create the Data Frame

We need to adjust for missing values in columns. This step will create the data frame as each key was a row and like this, the missing values are actually missing columns which is no problem for pandas (only missing rows lead to ValueError during creation). We need to transpose the data frame (flip the axis) and change the rows to columns. We also need to clean some of the data.

# Create a new dataframe. df = pd.DataFrame.from_dict(reviews_dict, orient='index') df.head()

# Delete all the columns that have missing values. df.dropna(axis=1, inplace=True) df.head()

# Transpose the dataframe. prod_reviews = df.T print(prod_reviews.head(10))

# Remove special characters from review text. prod_reviews['Reviews'] = prod_reviews['Reviews'].str.replace('\n','') prod_reviews.head(5)

prod_reviews.shape

# Convert dataframe to CSV file. prod_reviews.to_csv('dataset/reviews.csv', index=False, header=True)

Sentiment Analysis

Sentiment Analysis is a use case of Natural Language Processing (NLP) and comes under the category of text classification. To put it simply, Sentiment Analysis involves classifying a text into various sentiments, such as positive or negative, Happy, Sad or Neutral, etc. Thus, the ultimate goal of sentiment analysis is to decipher the underlying mood, emotion, or sentiment of a text. This is also known as Opinion Mining

#pip install transformers

#import transformers

from textblob import TextBlob

data=TextBlob(prod_reviews.Reviews[5]) print(data.sentiment) print(prod_reviews.Reviews[5]) if 0<data.sentiment[0]<0.5 : print('Neutral')

We can see the output is categorized between two — Polarity and Subjectivity.

Polarity is a float value within the range [-1.0 to 1.0] where 0 indicates neutral, +1 indicates a very positive sentiment and -1 represents a very negative sentiment.

Subjectivity is a float value within the range [0.0 to 1.0] where 0.0 is very objective and 1.0 is very subjective. Subjective sentence expresses some personal feelings, views, beliefs, opinions, allegations, desires, beliefs, suspicions, and speculations where as Objective sentences are factual.

prod_reviews['Review Type']=0 for i in range(len(prod_reviews)): data=TextBlob(prod_reviews.Reviews[i]) if data.sentiment[0]>=0.5: prod_reviews['Review Type'][i]='Positive' if 0<=data.sentiment[0]<0.5 : prod_reviews['Review Type'][i]='Neutral' if data.sentiment[0]<0 : prod_reviews['Review Type'][i]='Negative'

prod_reviews

res=pd.DataFrame(prod_reviews['Review Type'].value_counts()) res.plot.pie(y='Review Type',autopct='%1.1f%%',title="Review Types")