Frequent Statistics Report

import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import statsmodels.api as sm from scipy.stats import probplot import scipy.stats as stats import statsmodels.api as sm from scipy.stats import chi2_contingency sns.set_style("whitegrid") data = pd.read_csv('House_Rent_Dataset.csv')

#Making displot to show the whole population, and the sample of that population plt.figure(figsize=(24,24)) sns.displot(x=data['Rent'], kde=False, rug=False, bins = 100) plt.show()

#Checking response variable normality - rent prices filtered_rent_data = data[(data['Rent'] >= 0) & (data['Rent'] <= 30000)] fig, ax = plt.subplots(2,1,figsize=(12,12)) stats.probplot(filtered_rent_data['Rent'], plot=ax[0]) sns.distplot(filtered_rent_data['Rent']) plt.show()

As shown by the figures above, the response variable, rent prices, is observed to be slightly normally distributed. The QQ-plot suggest that there are some deviations from the line, especially at the beginning, however the data later begins to settle more closely to the line. The histogram suggest that there is a bell curve shape, as seen in normal distribution. Although, the graph does seem to be more right skewed.

fig, ax = plt.subplots(8, 1, figsize=(5, 20)) sns.histplot(filtered_rent_data['Size'], bins=20, ax=ax[0]) ax[0].set_xlabel('Size') ax[0].set_ylabel('Frequency') ax[0].set_title('Rental Property Size Analysis') categorical_vars = ['BHK', 'Tenant Preferred', 'City', 'Furnishing Status', 'Bathroom', 'Point of Contact'] for i, var in enumerate(categorical_vars, start=1): sns.countplot(x=var, data=filtered_rent_data, ax=ax[i]) ax[i].set_xlabel(var) ax[i].set_ylabel('Frequency') ax[i].set_title(f'Rental Property {var} Analysis') filtered_rent_data_copy = filtered_rent_data.copy() filtered_rent_data_copy['Posted On'] = pd.to_datetime(filtered_rent_data_copy['Posted On']) filtered_rent_data_copy['Month_Posted'] = filtered_rent_data_copy['Posted On'].dt.month sns.countplot(x='Month_Posted', data=filtered_rent_data_copy, ax=ax[7]) ax[7].set_xlabel('Month_Posted') ax[7].set_ylabel('Frequency') ax[7].set_title('Rental Property Posting Dates Analysis') plt.tight_layout() plt.show()

The univariate analysis illustrates that, Size, BHK and Bathroom are right skewed. City is well distributed amongst the data set. Furnishing status has the least frequent category as furnished rental places. The majority of properties opt for Bachelors/Family, and most properties are directly contacted through their owner. Finally, most properties are posted between May and June. With this in mind we can begin to fit the model and investigate which of these variables will have an influence in the renting prices observed.

Initial Model

#Creating dummy variables dummy_month = pd.get_dummies(filtered_rent_data_copy['Month_Posted'], prefix='Month' ) dummy_bhk = pd.get_dummies(filtered_rent_data['BHK'], prefix='BHK') dummy_poc = pd.get_dummies(filtered_rent_data['Point of Contact'], prefix='Point of Contact') dummy_city = pd.get_dummies(filtered_rent_data['City'], prefix='City') dummy_bathroom = pd.get_dummies(filtered_rent_data['Bathroom'], prefix='Bathroom') dummy_tenant_pref = pd.get_dummies(filtered_rent_data['Tenant Preferred'], prefix='Tenant Preferred') dummy_furnishing_status = pd.get_dummies(filtered_rent_data['Furnishing Status'], prefix='Furnishing Status') X = pd.concat([filtered_rent_data[['Size']], dummy_bhk, dummy_poc, dummy_city, dummy_bathroom, dummy_tenant_pref, dummy_furnishing_status, dummy_month], axis=1) y = filtered_rent_data['Rent'] X = sm.add_constant(X) model = sm.OLS(y, X) results = model.fit() print(results.summary())

Final model

X = X.drop(['BHK_6', 'BHK_2','Point of Contact_Contact Builder', 'Point of Contact_Contact Owner','Bathroom_2', 'Bathroom_6', 'Bathroom_5', 'Bathroom_4','City_Bangalore','Bathroom_1','Tenant Preferred_Bachelors/Family','Month_6','Furnishing Status_Semi-Furnished','Month_5','Tenant Preferred_Family','Month_4','Bathroom_3','BHK_1','City_Chennai'], axis=1) model = sm.OLS(y, X) results = model.fit() print(results.summary())

Final Correlation Matrix

X = pd.concat([filtered_rent_data[['Rent','Size']], dummy_bhk, dummy_poc, dummy_city, dummy_bathroom, dummy_tenant_pref, dummy_furnishing_status, dummy_month], axis=1) X = X.drop(['BHK_6', 'BHK_2','Point of Contact_Contact Builder', 'Point of Contact_Contact Owner','Bathroom_2', 'Bathroom_6', 'Bathroom_5', 'Bathroom_4','City_Bangalore','Bathroom_1','Tenant Preferred_Bachelors/Family','Month_6','Furnishing Status_Semi-Furnished','Month_5','Tenant Preferred_Family','Month_4','Bathroom_3','BHK_1','City_Chennai'], axis=1) corr = X.corr() corr.style.background_gradient(cmap='coolwarm')

ypredicted = results.predict() residuals = filtered_rent_data['Rent'] - ypredicted fig,ax = plt.subplots(6,2, figsize = (20,80)) sns.scatterplot(y = ypredicted, x = np.abs(residuals), ax=ax[0,0]) #scatterplot of abs residual vs predicted ax[0,0].set_title('Absolute vs. Predicted Residuals') ax[0,0].set_ylabel('residuals') ax[0,0].set_xlabel('predictions') stats.probplot(residuals, plot=ax[0,1]) ax[0,1].set_title('Residuals') sns.distplot(residuals, ax=ax[1,0]) ax[1,0].set_title('Residuals') ax[1,0].set_xlabel('residuals') x_axis = range(len(filtered_rent_data)) sns.scatterplot(x = x_axis, y = residuals, ax=ax[1,1]) ax[1,1].set_title('Order of Collection vs. Residuals') ax[1,1].set_ylabel('residuals') ax[1,1].set_xlabel('order of collection') sns.scatterplot(x = filtered_rent_data['Size'], y = residuals, ax=ax[2,0]) ax[2,0].set_title('Size vs Residuals') ax[2,0].set_ylabel('residuals') ax[2,0].set_xlabel('size') sns.stripplot(x = filtered_rent_data['BHK'], y = residuals, ax=ax[2,1]) ax[2,1].set_title('BHK vs Residuals') ax[2,1].set_ylabel('residuals') ax[2,1].set_xlabel('BHK') sns.stripplot(x = filtered_rent_data['Point of Contact'], y = residuals, ax=ax[3,0]) ax[3,0].set_title('Point of Contact vs Residuals') ax[3,0].set_ylabel('residuals') ax[3,0].set_xlabel('point of contact') sns.stripplot(x = filtered_rent_data['City'], y = residuals, ax=ax[3,1]) ax[3,1].set_title('City vs Residuals') ax[3,1].set_ylabel('residuals') ax[3,1].set_xlabel('city') sns.stripplot(x = filtered_rent_data['Tenant Preferred'], y = residuals, ax=ax[4,0]) ax[4,0].set_title('Tenant Preference vs Residuals') ax[4,0].set_ylabel('residuals') ax[4,0].set_xlabel('bachelors') sns.stripplot(x = filtered_rent_data['Furnishing Status'], y = residuals, ax=ax[4,1]) ax[4,1].set_title('Furnishing Status vs Residuals') ax[4,1].set_ylabel('residuals') ax[4,1].set_xlabel('furnishing status') sns.stripplot(x = filtered_rent_data_copy['Month_Posted'], y = residuals, ax=ax[5,0]) ax[5,0].set_title('Month 7 vs Residuals') ax[5,0].set_ylabel('residuals') ax[5,0].set_xlabel('july')

The first diagram, which depicts the Absolute vs. Predicted Residuals, shows how the variability is somewhat constant as most of the data is found within the same general area in both axes. The QQ-plot and the histogram for the residuals shows that the data is normally distributed. As the points in the QQ-Plot follow the line and the Histogram has a bell-shaped curve distribution. The residuals have also been visualized to be independent as the Order of Collection vs. Residuals diagram shows no trend in the data. The rest of the diagrams demonstrate their linearity related to the outcome and the explanatory variable themselves. Since all the assumptions of the model are not infringed this means, the model has been fit adequately, and no further transformations are needed.