CourseWork 2 - H00444598
Introduction
This study provides an in-depth analysis of predictive models developed to better understand the factors that influence profit within a dataset. Exploratory data analysis, feature engineering, and the use of Linear Regression and Random Forest Regressor models reveal insights into how variables like sales, quantity, and discounts affect profit. Regulatory approaches include modifying the profit variable to handle negative values and standardizing features to improve model performance. Furthermore, the report provides a visual representation of model performance, as well as the importance and influence of certain factors that contribute to improving profit prediction. The analysis serves as the foundation for strategic decision-making, to increase profitability through informed, data-driven insights.
Importing Data
import pandas as pd
df = pd.read_csv('dataset')
df
Data cleaning
In the data-cleaning process for this analysis, the following steps have been undertaken:
Elimination of Outliers: After addressing missing values, duplicates, and formatting date & time, outliers in Sales, Profit, and Quantity columns were identified and capped using the Interquartile Range (IQR) method. This aimed to mitigate the influence of extreme values on predictions, potentially reducing high Mean Squared Error (MSE) values.
Feature Engineering: A new feature, "Sales/Quantity," was introduced to capture variations in the relationship between sales and product quantity. This consolidation of information from two variables is expected to enhance predictive accuracy by simplifying the model and emphasizing significant signals.
Transformation of the Profit Variable: A logarithmic adjustment was applied to the Profit variable to address its skewed distribution and negative values. This transformation normalized the distribution and efficiently handled negative values, making it more compatible with models such as linear regression.
Feature Scaling: StandardScaler from sklearn was employed to normalize features, ensuring each feature contributes equally to the model by scaling them to a mean of zero and a standard deviation of one. This process enhances training optimization and overall model accuracy.
These cleaning and preprocessing steps are essential for optimizing the dataset for modeling, helping to enhance model performance and ensure more accurate predictions.
Checking for missing Values
df.isnull().sum()
Checking for Duplicates
df.duplicated().sum()
Correcting all the data types
df.info()
#Correcting Date Format
df['Order Date'] = pd.to_datetime(df['Order Date'], dayfirst=True)
df['Ship Date'] = pd.to_datetime(df['Ship Date'], dayfirst=True)
df.dtypes
Removing Extreme Outliers
# Removing extreme outliers with defined condition for 'Quantity', 'Discount', 'Sales', 'Profit'
extreme_outliers_df = pd.DataFrame()
filtered_df = df.copy()
cols_to_check = ['Quantity', 'Discount', 'Sales', 'Profit']
for col in cols_to_check:
Q1 = filtered_df[col].quantile(0.25)
Q3 = filtered_df[col].quantile(0.75)
IQR = Q3 - Q1
extreme_lower_bound = Q1 - 3 * IQR
extreme_upper_bound = Q3 + 3 * IQR
# Filter out the extreme outliers
condition = (filtered_df[col] < extreme_lower_bound) | (filtered_df[col] > extreme_upper_bound)
extreme_outliers_df = pd.concat([extreme_outliers_df, filtered_df[condition]])
filtered_df = filtered_df[~condition]
# Removing duplicates from extreme_outliers_df as some rows may be flagged as extreme outliers in multiple columns
extreme_outliers_df = extreme_outliers_df.drop_duplicates()
len(extreme_outliers_df)
filtered_df.shape
Summary of the Statistics
summary_statistics = filtered_df.describe()
summary_statistics
import matplotlib.pyplot as plt
# Summary of dataset before and after outlier removal
original_size = df.shape[0]
cleaned_size = filtered_df.shape[0]
removed_records = original_size - cleaned_size
# Preparing data for the bar chart
categories = ['Original', 'Cleaned', 'Removed']
sizes = [original_size, cleaned_size, removed_records]
colors = ['skyblue', 'lightgreen', 'salmon']
plt.figure(figsize=(10, 6))
bars = plt.bar(categories, sizes, color=colors)
# Adding labels directly to each bar
for bar in bars:
yval = bar.get_height()
plt.text(bar.get_x() + bar.get_width()/2, yval + 3, yval, ha='center', va='bottom')
plt.title('Dataset Size: Before and After Outlier Removal')
plt.ylabel('Number of Records')
plt.show()
Summary
The summary statistics of the dataset after removing 1546 outliers based on extreme conditions for `Quantity`, `Discount`, `Sales`, and `Profit` provide insightful details:
Sales: The average (mean) sales amount is approximately $77.86, suggesting a typical transaction size in the dataset. The data varies widely though, as evidenced by a standard deviation of $97.26, indicating that while many transactions hover around the mean, there are quite a few that are significantly higher or lower. The range of sales from the minimum of $0.444 to a maximum of $783.96, after outlier removal, shows a constrained transaction space, likely removing extremely high-value sales that might skew analysis.
Quantity: The average quantity sold per transaction stands at 3.56, with transactions varying mostly by 2.08 units around this mean (standard deviation). This indicates that the majority of transactions involve the purchase of a few items, with the range being constrained from 1 to 14 items after cleaning the data from extreme outliers.
Discount: The mean discount provided across transactions is about 15.1%, with a variation (standard deviation) of roughly 20.2%, indicating some transactions had more substantial discounts while others had none (0%). The maximum discount observed is 80%, which suggests that even after removing outliers, the dataset contains transactions with substantial discounts.
Profit: The average profit from transactions is $12.49, showing what might be expected from typical transactions. However, the standard deviation is $23.19, indicating that profits vary considerably across transactions. After cleaning, profits still range from a loss of $64.54 to a gain of $89.96, highlighting the existence of both loss-making and highly profitable transactions even after filtering extreme outliers.
Returned & Not Returned: Within the cleaned dataset, it was discerned that 8.1% of products were returned by customers, while 91.9% were not. This data suggests general customer satisfaction with the product quality and their expectations, with a significant majority opting to keep their purchases.
These cleaned summary statistics suggest a more normalized dataset by trimming the data points that are exceedingly far from the median, which allows for more accurate analysis and modeling without the undue influence of extreme values.
Bar Charts for Profitability
plt.figure(figsize=(15, 10))
# Total Profit by Category
df.groupby('Category')['Profit'].sum().plot(kind='bar')
plt.title('Total Profit by Category')
plt.xlabel('Category')
plt.ylabel('Total Profit')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
The generated graph presents a clear visual analysis of all the categories of different products contributing to the total profit within the dataset.
Total Profit by Category: The analysis reveals disparities in profitability among product categories with "Technology" leading in contributing the most to profitability. This is followed by "Office Supplies," which occupies the second position, and "Furniture" trails as the category with the least contribution to overall profits. This ranking underscores the significant role the "Technology" sector plays in driving the company's profit margins.
Scatter Plot:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
df = pd.read_csv('dataset')
df['Sales/Quantity'] = df['Sales'] / df['Quantity']
plt.figure(figsize=(10, 6))
sns.scatterplot(data=df, x='Sales', y='Profit', alpha=0.5)
plt.title('Sales vs Profit Scatter Plot')
plt.xlabel('Sales')
plt.ylabel('Profit')
plt.show()
Sales vs. Profit:
The scatter plot "Sales vs Profit" visually represents the relationship between sales and profit:
Generally, as sales increase, profit tends to rise, but this trend isn't perfectly linear.
Outliers, particularly at higher sales levels, suggest that increased sales may not always result in proportionate profit gains.
There are instances of both significant profit and loss across various sales amounts.
A cluster of data points with lower sales values indicates common occurrences of smaller transactions in business datasets.
The diversity in profit levels at lower sales ranges ranges from losses to substantial profits.
Notably, there's at least one data point with high sales but a loss, suggesting a unique case worthy of further investigation.
This plot serves as a valuable tool for identifying trends, pinpointing outliers, and informing decisions related to sales strategies and product profitability.
Regression Model:
Linear Regression Model and Random Forest Regressor
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.preprocessing import StandardScaler
df['Sales/Quantity'] = df['Sales'] / df['Quantity']
# Transform Profit to handle negative values effectively
df['Profit'] = np.log(df['Profit'] + df['Profit'].abs().max() + 1)
features = df[['Sales', 'Quantity', 'Discount', 'Sales/Quantity']]
target = df['Profit']
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
lm = LinearRegression()
lm.fit(X_train_scaled, y_train)
y_pred = lm.predict(X_test_scaled)
# Calculating MSE and R2 score for Linear Regression
mse_lm = mean_squared_error(y_test, y_pred)
r2_lm = r2_score(y_test, y_pred)
# Calculating Adjusted R-squared for Linear Regression
n = X_test.shape[0]
p = X_test.shape[1]
adj_r2_lm = 1 - (1-r2_lm) * (n-1) / (n-p-1)
# Coefficients for Linear Regression
coefficients_lm = lm.coef_
# Random Forest Regressor Model
rfr = RandomForestRegressor(random_state=42)
rfr.fit(X_train_scaled, y_train)
y_pred_rfr = rfr.predict(X_test_scaled)
# Calculating MSE and R2 score for Random Forest Regressor
mse_rfr = mean_squared_error(y_test, y_pred_rfr)
r2_rfr = r2_score(y_test, y_pred_rfr)
# Calculating Adjusted R-squared for Random Forest Regressor
adj_r2_rfr = 1 - (1-r2_rfr) * (n-1) / (n-p-1)
# Feature Importances from Random Forest Regressor
feature_importances_rfr = rfr.feature_importances_
(mse_lm, r2_lm, adj_r2_lm, coefficients_lm, mse_rfr, r2_rfr, adj_r2_rfr, feature_importances_rfr)
import matplotlib.pyplot as plt
import seaborn as sns
# Visualization of Linear Regression Model Performance
plt.figure(figsize=(10, 6))
plt.scatter(y_test, y_pred, color='blue', alpha=0.5, label='Predicted vs Actual')
plt.plot(y_test, y_test, color='red', label='Actual')
plt.title('Linear Regression: Predicted vs Actual Profit')
plt.xlabel('Actual Profit')
plt.ylabel('Predicted Profit')
plt.legend()
plt.show()
#Visualization of Random Forest Regressro Model Perofromance
plt.figure (figsize=(10, 6))
plt.scatter(y_test, y_pred, color='blue', alpha=0.5, label='Predicted vs Actual')
plt.plot(y_test, y_test, color='red', label='Actual')
plt.title ('Random Forest Regressor: Predicted vs Acutal Profit')
plt.xlabel('Actual Profit')
plt.ylabel('Predicted Profit')
plt.legend()
plt.show()
# Visualization of feature importances for Linear Regression and Random Forest Regressor
# Create subplots
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(14, 6))
# Linear Regression Coefficients
sns.barplot(x=coefficients_lm, y=['Sales', 'Quantity', 'Discount', 'Sales/Quantity'], ax=axs[0])
axs[0].set_title('Linear Regression Coefficients')
# Random Forest Feature Importances
sns.barplot(x=feature_importances_rfr, y=['Sales', 'Quantity', 'Discount', 'Sales/Quantity'], ax=axs[1])
axs[1].set_title('Random Forest Feature Importances')
plt.tight_layout()
plt.show()
Adopted Approaches for Modeling
Feature Engineering:
The creation of a new feature “Sales/Quantity” is aimed at capturing the efficiency of sales in relation to the quantity. This potentially useful metric isn't directly observable from the original features but could significantly influence the profit. By including this, we aim to enrich the model with insights that raw sales and quantity figures alone cannot provide. Similarly, the log transformation of the `Profit` feature addresses skewness in its distribution, enhancing the model's ability to capture underlying patterns and relationships.
Data Scaling:
The use of “StandardScaler” to standardize the features facilitates the training of machine learning models. The features are standardized to have a mean of 0 and a standard deviation of 1. Many algorithms, including those used here, can be sensitive to the scale of input variables. Features on vastly different scales can distort the underlying patterns they intend to model, especially in distance-based methods (though less so in tree-based methods like Random Forest). Standardizing ensures that each feature contributes equally to the model's predictions.
A logarithmic transformation “Profit”:
This transformation was necessary for stabilizing the variance and making the data more 'normal' in distribution, which enhanced the models’ performance, especially the linear one. Additionally, the transformation deals with negative profit values by shifting all values up before taking the log
Model Selection
Linear Regression and Random Forest Regressor: These models are chosen for their complementary strengths. Linear Regression, a simple model, offers transparency and fast training times, making it a natural starting point. It's best suited for datasets where relationships between variables are linear. On the other hand, Random Forest, an ensemble of decision trees, can capture non-linear relationships and interactions between features without explicit feature engineering, offering a more powerful, if less interpretable, alternative.
The evaluation metrics and feature importances for both Linear Regression and Random Forest Regressor models offer insight into the dataset's characteristics and the models' performance:
Linear Regression:
MSE: “0.00028486” This value indicates the model's average squared difference between the estimated values and the actual value. The closer this number is to 0, the better the model's performance. In this context, the value suggests a decent fit.
R²: “0.21165” This score shows that approximately 21.17% of the variance in the profit can be explained by the model's inputs. This value is relatively low, indicating a modest fit to the data.
Adjusted R²: “0.21007” This adjusted metric, slightly lower than the R², accounts for the number of predictors in the model, affirming the modest explanatory power of the model.
Coefficients: “0.00516, 0.00008, -0.00632, 0.00546” These coefficients represent the change in the profit for a one-unit change in each feature, holding the other features constant. The negative value for the 'Discount' feature suggests that an increase in discounts tends to decrease the profit, while 'Sales', 'Quantity', and the engineered 'Sales/Quantity' ratio have a positive relationship with profit.
Random Forest Regressor:
MSE: “0.00016105” This model achieved a lower MSE compared to Linear Regression, indicating a closer fit to the actual data.
R²: “0.55428” The R² value significantly improves in the Random Forest model, with over 55% of the variance in profit being explained by the features, suggesting a good fit.
Adjusted R²: “0.55339” Similar to the R², this adjusted metric is also substantially better than that of the Linear Regression model, signaling a more accurate representation of the dataset's characteristics.
Feature Importances: “62.86%, 2.82%, 22.48%, 11.84%” These values illustrate the relative importance of each feature in predicting the profit. 'Sales' is the most influential feature, followed by 'Discount', 'Sales/Quantity', and 'Quantity', respectively. This indicates that while sales volume is a significant driver of profit, discounting practices and the efficiency of sales (as captured by the 'Sales/Quantity' ratio) also play critical roles.
Recommendations
Based on the analysis performed using Linear Regression and Random Forest Regressor models, as well as the insights from the visualization of sales vs. profit, here are some recommendations that could potentially improve profitability and operational strategies within the dataset context:
Optimize Discount Strategy: The findings suggest a nuanced relationship between discounts and profitability. It might be useful to conduct a deeper analysis to identify an optimal discounting strategy that maximizes sales without significantly reducing profit margins. This could involve setting discount thresholds that are enticing to customers yet still preserve profitability (Gregorutti et al., 2016).
Focus on High-Profit Categories: The analysis indicates that not all product categories contribute equally to profits, with the technology category showing promising results. It is recommended to prioritize marketing and resource allocation toward high-profit categories, potentially increasing the company's overall profitability.
Regional Optimization: With the analysis revealing variations in profitability across regions, a tailored approach to each region could enhance performance. Strategies might include focusing on marketing efforts in high-profit regions, optimizing distribution channels, or even reviewing product offerings based on regional preferences.
Efficient Sales Strategies: The engineered feature 'Sales/Quantity' ratio indicated its significance in predicting profit. Strategies that enhance the efficiency of sales relative to the quantity sold should be considered. This might involve bundling products, upselling, or providing incentives for larger purchases.
Shipping Method Analysis: The correlation between shipping modes and profitability suggests re-evaluating the cost-effectiveness of different shipping options. It might be beneficial to promote more profitable shipping options to customers or revisit pricing strategies for less profitable modes to better balance customer satisfaction with profitability.
Conclusion
The Random Forest Regressor model outperforms the Linear Regression model in terms of both fit and predictive accuracy, as indicated by lower MSE and higher R² values. This suggests that the dataset likely contains non-linear relationships and interactions between features that the Random Forest model can capture but the Linear Regression model cannot (Grömping, 2009).
The negative impact of discounts on profit as suggested by the Linear Regression coefficients, coupled with its significant feature importance in the Random Forest model, points to a nuanced relationship between discounting practices and profitability.
The pivotal role of 'Sales' in driving profit, as revealed by its feature importance in the Random Forest model, underscores the importance of sales volume in overall profitability. However, the importance of the 'Sales/Quantity' ratio suggests that not just volume, but the efficiency of sales relative to the quantity sold, is crucial for maximizing profit.
These findings suggest areas for strategic focus, such as optimizing discounting strategies and focusing on high sales volume while considering the efficiency of those sales in contributing to profit.
References
Grömping, U. (2009) ‘Variable importance assessment in regression: Linear regression versus Random Forest’, The American Statistician, 63(4), pp. 308–319. doi:10.1198/tast.2009.08199.
Gregorutti, B., Michel, B. and Saint-Pierre, P. (2016) ‘Correlation and variable importance in random forests’, Statistics and Computing, 27(3), pp. 659–678. doi:10.1007/s11222-016-9646-1.
Ertekin, N., Shulman, J.D. and Chen, H. (Allan) (2019) ‘On the profitability of stacked discounts: Identifying revenue and cost effects of discount framing’, Marketing Science, 38(2), pp. 317–342. doi:10.1287/mksc.2018.1137.