Data Analytics with Pandas 🐼
The Pandas package is the most imperative tool in Data Science and Analysis working in Python nowadays.
In this notebook, we would analyze Supermarket Data Across the Country; Company XYZ.
Alright, let’s start!
Step 1 - Loading the Dataset
import os
import glob
import pandas as pd
import numpy as np
path ="/work"
os.chdir(path) #Current working directory that contains your dataset and code file
#Use glob to match the pattern "csv"
extension = 'csv'
result = glob.glob('*.{}'.format(extension))
print (result)
['Lagos_Branch.csv', 'Port_Harcourt_Branch.csv', 'Abuja_Branch.csv']
dir = "./"
csv_files = []
for file in glob.iglob(dir + "/*."+extension, recursive=True):csv_files.append(file)
#Combine all the files in the generated list above and export to a CSV
df = []
for file in csv_files:
dff = pd.read_csv(file)
df.append(dff)
branch = pd.concat(df, axis = 0)
branch.to_csv("Branch.csv")
df = pd.read_csv('Branch.csv')
Step 2 - Data Exploration
It is an important pillar of data science, a critical step required to complete every project regardless of the domain or the type of data you are working with.
The dataset was explored using some built-in pandas function like; Numpy for carrying out numerical computations, pandas for making a dataframe object seaborn and matplotlib for visualizations. The .head() to view the first few observations of the dataframe and .info() for information about the dataframe.
# Import Libaries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('fivethirtyeight')
import warnings
warnings.filterwarnings('ignore')
df.head(3)
#Check the number of rows and columns present in the data using the shape attribute.
df.shape
#are there missing values
df.isna().sum()
#the names of the columns using the columns attribute.
df.columns
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 18 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 1000 non-null int64
1 Invoice ID 1000 non-null object
2 Branch 1000 non-null object
3 City 1000 non-null object
4 Customer type 1000 non-null object
5 Gender 1000 non-null object
6 Product line 1000 non-null object
7 Unit price 1000 non-null float64
8 Quantity 1000 non-null int64
9 Tax 5% 1000 non-null float64
10 Total 1000 non-null float64
11 Date 1000 non-null object
12 Time 1000 non-null object
13 Payment 1000 non-null object
14 cogs 1000 non-null float64
15 gross margin percentage 1000 non-null float64
16 gross income 1000 non-null float64
17 Rating 1000 non-null float64
dtypes: float64(7), int64(2), object(9)
memory usage: 140.8+ KB
#Statiscal Summary
df.describe()
Step 3 - Dealing with DateTime Features
import datetime as dt #Importing our datetime module
df["Date"] = pd.to_datetime(df["Date"]) # Extracting the Dates
df.dtypes
df["Time"] = pd.to_datetime(df["Time"]) #Extracting Time
df["Day"] = df["Date"].dt.day
df["Month"] = df["Date"].dt.month
df["Hour"] = df["Time"].dt.hour
df["Year"] = df["Date"].dt.year
df["Hour"].nunique()
df["Hour"].unique()
df.columns
Step 4 - Unique Values in Columns
categorical_columns = [col for col in df.columns if df[col].dtype == "object" ]
categorical_columns
df['Branch'].unique().tolist()
print("Total Number of unique values in the Branch Column : {}". format(len(df['Branch'].unique().tolist())))
Total Number of unique values in the Branch Column : 3
for i in categorical_columns:
if i != 'Invoice ID':
print ("This is for {} Coulmns".format(i))
print (df[i].unique().tolist())
This is for Branch Coulmns
['A', 'C', 'B']
This is for City Coulmns
['Lagos', 'Port Harcourt', 'Abuja']
This is for Customer type Coulmns
['Member', 'Normal']
This is for Gender Coulmns
['Female', 'Male']
This is for Product line Coulmns
['Health and beauty', 'Home and lifestyle', 'Sports and travel', 'Electronic accessories', 'Food and beverages', 'Fashion accessories']
This is for Payment Coulmns
['Epay', 'Card', 'Cash']
for i in categorical_columns:
if i != 'Invoice ID':
print ("This is for {} Coulmns".format(i))
print ("Total Number of unique value: {}". format(len(df[i].unique().tolist())))
This is for Branch Coulmns
Total Number of unique value: 3
This is for City Coulmns
Total Number of unique value: 3
This is for Customer type Coulmns
Total Number of unique value: 2
This is for Gender Coulmns
Total Number of unique value: 2
This is for Product line Coulmns
Total Number of unique value: 6
This is for Payment Coulmns
Total Number of unique value: 3
Step 5 - Aggregation with GroupBy
# Creating a groupby object to group the dataset according to City
by_city = df.groupby("City")
#To get the sum and mean of the numeric columns
by_city.agg(["sum", "mean"])
# Using the groupby object ot get the gross income of each city
gross_city = by_city['gross income'].sum().reset_index()
gross_city
#Getting the city with the highest gross income
gross_city.max().reset_index()
City_with_others = by_city["Unit price", "Quantity", "gross margin percentage"].sum()
City_with_others
#Product line with the highest gross income
prod = df[['Product line', 'gross income']]
prod_group = prod.groupby('Product line').sum().reset_index().sort_values(['gross income'], ascending =False)
prod_group
sales_quantity = df[['Branch','Quantity']]
group = sales_quantity.groupby('Branch').count().sort_values(['Quantity'], ascending =False)
group
Step 6 - Data Visualization
sns.countplot(x= 'Branch', data = df).set_title("Branch vs Sales", size = 24);
sns.countplot(x = 'Payment', data = df).set_title('Different payment card', size = 23);
sns.countplot(x = 'City', data = df).set_title('City Vs Sales');
sns.countplot(y = 'Product line', data = df).set_title('Sales in different product line');
sns.countplot(y = 'Product line', data = df, hue = 'Payment').set_title('Payment mode VS Product line');
sns.countplot(x = 'Payment', data = df, hue = 'Branch').set_title('Payment mode VS Branch')
plt.legend(loc='upper left', bbox_to_anchor=(1.25, 0.5), ncol=1)
sns.boxplot(x = 'Branch', y = 'Rating', data = df).set_title("Rating vs Branch");
sns.catplot(x = 'Product line', y = 'Quantity', hue = 'Gender', kind= 'boxen', data = df, aspect = 4);
sns.catplot(y = 'Unit price', x = 'Product line', kind= 'point', data = df, aspect = 4);
sns.catplot(y = 'Quantity', x = 'Product line', kind= 'point', data = df, aspect = 4);
sns.countplot(x = 'Payment', data = df, hue = 'Gender').set_title("Gender vs Payment method");
sns.catplot(y = 'Tax 5%', x = 'City', kind= 'swarm', data = df, aspect =4);
sns.countplot(x = 'City', hue = 'Gender', data = df).set_title("Gender VS City");
Summary
In this tutorial, we have learnt how to perform Exploratory Data Analysis (EDA) in Python and how to use external Python packages such as Pandas, Numpy, Matplotlib, Seaborn etc. to conduct univariate analysis, bivariate analysis and data visualization.
from PIL import Image
from PIL import GifImagePlugin
imageObject = Image.open("tenorr.gif")
size = (500,300)
imageObject = imageObject.resize(size)
imageObject.show()