Final Project: Looking for Airbnbs in Los Angeles?
By: Krtin Juneja and Elizabeth Veilleux
Abstract:
This project focuses on data about Airbnbs in Los Angeles. We both love to travel and the next area we want to travel to is Los Angeles. We are searching for an Airbnb in the Los Angeles area that we can rent. Since we are college students we tend to not have the biggest income coming in so we want to find an Airbnb that is rather cheap. This led us to thinking about what factors play a role in Airbnb prices increasing. Is it the amount of bedrooms? How about the average rent price of the area? Or maybe just the area or room type? We want to determine this with the data we currently have available to us. SO, we will be running a multiple linear regression to see what does contribute to the price of an Airbnb in Los Angeles along with creating an Airbnb price predictor with machine learning. Additionally, we will see if certain neighborhoods tend to have significantly higher prices.
About the Datasets:
We have a dataset dealing with Los Angeles Airbnb data. The listings dataset gives various details about the Airbnb rental itself along with host information. Additionally, we have a dataset with rent prices for LA neighborhoods from 2010-2016 (we will be focusing on just 2016 data). This rent prices dataset is useful because it provides rent amounts for LA neighborhoods which we can utilize as a predictor since it gives some insight on how real estate and cost is normally for the area. However, it is important to note that the Airbnb data is from 2020-2021 so there will be some discrepancies as the rent would definitely be hire in 2020-2021 due to factors like inflation.
GOAL: To see what are the best predictors of Airbnb rental prices in Los Angeles and if certain neighborhoods are more pricey.
Loading in the Data:
# Import pandas
import pandas as pd
# Read listings csv file in
listings = pd.read_csv('listings.csv')
# Get an overview of data
listings.head()
# Read rent prices csv file in
Rent_LA = pd.read_csv('Rent_Price__LA_.csv')
# Get an overview of data
Rent_LA.tail()
Cleaning the data:
# Subset to get only columns we want
listings_cleaned = listings[listings['accommodates']>0][['id', 'neighbourhood_cleansed','accommodates', 'bedrooms', 'beds','room_type', 'price']]
# See an overviews of cleaned data so far
listings_cleaned.head()
# Create a Price Cleaning function
def price_cleaner(price):
price = price.replace('$','')
price = float(price.replace(',',''))
return price
# Apply the function to our data
listings_cleaned['price'] = listings_cleaned['price'].apply(price_cleaner)
# Overview of the data
listings_cleaned.head()
# Room Type Dummy Variables
listings_cleaned['Entire Property']=listings_cleaned['room_type'].apply(lambda x: 1 if x == 'Entire home/apt' else 0)
listings_cleaned['Private Room']=listings_cleaned['room_type'].apply(lambda x: 1 if x == 'Private room' else 0)
listings_cleaned['Hotel Room']=listings_cleaned['room_type'].apply(lambda x: 1 if x == 'Hotel room' else 0)
listings_cleaned['Shared Room']=listings_cleaned['room_type'].apply(lambda x: 1 if x == 'Shared room' else 0)
# Overview of data
listings_cleaned.head()
# Clean and subset Rent prices data
Rent_LA_cleaned = Rent_LA[Rent_LA['Year']==2016][['Neighborhood', 'Amount']].dropna().drop_duplicates('Neighborhood').reset_index()
Rent_LA_cleaned['Amount'] = Rent_LA_cleaned['Amount'].str.replace(',','').astype(float)
# Merge two dataframes
listings_cleaned = listings_cleaned.merge(Rent_LA_cleaned, left_on='neighbourhood_cleansed', right_on='Neighborhood', how='left').drop(['index','Neighborhood'], axis = 1)
# Overview of merged data
listings_cleaned.head()
Multiple Linear Regression:
# Import necessary packages
import seaborn as sns
import matplotlib.pyplot as plt
# Make pairplot
sns.pairplot(data = listings_cleaned[['accommodates','bedrooms', 'beds', 'price','Amount']])
plt.show()
# Create a scatterplot function
def scatter_plot (column_x,data,title):
sns.set_style('darkgrid')
fig = sns.scatterplot(x = data[column_x],y = data['price'],data=data, color = 'red')
fig.set_title(title, fontsize=14)
fig.set(xlabel = column_x.capitalize(), ylabel = 'Airbnb Price')
plt.show()
# Run scatterplot function on accommodates
scatter_plot('accommodates',listings_cleaned,'Airbnb Price Vs Accommodates')
# Run scatterplot function on bedrooms
scatter_plot('bedrooms',listings_cleaned,'Airbnb Price Vs Bedrooms')
# Run scatterplot function on beds
scatter_plot('beds',listings_cleaned,'Airbnb Price Vs Beds')
# Run scatterplot function on monthly rent prices
scatter_plot('Amount',listings_cleaned,'Airbnb Price Vs Monthly Rent Prices for LA Neighborhoods')
# Create correlation heat map
sns.heatmap(listings_cleaned.corr(), annot=True)
plt.show()
# Creates series/list of correlation between predictors and price
listings_cleaned.corr()['price'].sort_values(ascending=False)
# Import needed packages
from sklearn import linear_model
import statsmodels.api as sm
# Create a variable that drops nan values for MLR
listings_dropna = listings_cleaned[['accommodates','bedrooms', 'beds', 'price', 'Entire Property', 'Private Room', 'Hotel Room', 'Shared Room','Amount']].dropna().astype(int)
# Name the predictor and response variables
X = listings_dropna[['accommodates','bedrooms', 'beds', 'Entire Property', 'Private Room', 'Hotel Room', 'Shared Room','Amount']]
Y = listings_dropna['price']
# Fit the model with sklearn
regr = linear_model.LinearRegression()
regr.fit(X, Y)
# Print the intercept and coefficients of the model with sklearn
print('Intercept: \n', regr.intercept_)
print('Coefficients: \n', regr.coef_)
# Add intercept/constant to model summary below with statsmodels
X = sm.add_constant(X)
# Create the model with statsmodels
model = sm.OLS(Y, X).fit()
# Print the model summary to discuss how promising the model is with statsmodels
print_model = model.summary()
print(print_model)
Intercept:
-382.5153626993822
Coefficients:
[ 3.77936532e+01 2.17147009e+02 -2.92019654e+01 -1.72431912e+02
-1.09265314e+02 3.60870110e+02 -7.91728846e+01 2.15727008e-01]
OLS Regression Results
==============================================================================
Dep. Variable: price R-squared: 0.239
Model: OLS Adj. R-squared: 0.239
Method: Least Squares F-statistic: 1219.
Date: Tue, 29 Jun 2021 Prob (F-statistic): 0.00
Time: 14:36:22 Log-Likelihood: -2.1001e+05
No. Observations: 27155 AIC: 4.200e+05
Df Residuals: 27147 BIC: 4.201e+05
Df Model: 7
Covariance Type: nonrobust
===================================================================================
coef std err t P>|t| [0.025 0.975]
-----------------------------------------------------------------------------------
const -306.0123 14.345 -21.332 0.000 -334.130 -277.894
accommodates 37.7937 2.577 14.667 0.000 32.743 42.844
bedrooms 217.1470 5.371 40.430 0.000 206.620 227.674
beds -29.2020 3.445 -8.478 0.000 -35.953 -22.451
Entire Property -248.9350 12.571 -19.803 0.000 -273.574 -224.296
Private Room -185.7684 12.380 -15.006 0.000 -210.034 -161.503
Hotel Room 284.3670 42.000 6.771 0.000 202.045 366.689
Shared Room -155.6760 18.272 -8.520 0.000 -191.490 -119.862
Amount 0.2157 0.008 28.718 0.000 0.201 0.230
==============================================================================
Omnibus: 48273.835 Durbin-Watson: 1.764
Prob(Omnibus): 0.000 Jarque-Bera (JB): 72218492.949
Skew: 12.808 Prob(JB): 0.00
Kurtosis: 254.340 Cond. No. 2.01e+18
==============================================================================
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 1.77e-26. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
Predicting Airbnb Price with Machine Learning
# Create a function to fit model
def fit_model_to(dataset):
predictors = dataset[['accommodates','bedrooms', 'beds', 'Entire Property', 'Private Room', 'Hotel Room', 'Shared Room','Amount']]
response = dataset['price']
model = linear_model.LinearRegression()
model.fit(predictors,response)
standardized = (predictors - predictors.mean())/ predictors.std()
standardized_model = linear_model.LinearRegression()
standardized_model.fit(standardized,response)
coefficients = pd.Series(standardized_model.coef_[0], index = predictors.columns).abs().sort_values()
print(coefficients)
return model
# Create a function to score model
def score_model(model, dataset):
predictors = dataset[['accommodates','bedrooms', 'beds', 'Entire Property', 'Private Room', 'Hotel Room', 'Shared Room','Amount']]
dataset['Prediction'] = model.predict(predictors).astype(int)
dataset['Correct'] = dataset['Prediction'] == dataset['price']
scatter_plot('Prediction',dataset,'Airbnb Price Vs Predicted Price')
precision = dataset['Correct'].sum()/len(dataset)
print(f"The proportionn of the predictions that were correct is: {precision}")
TP = ( dataset['price'] & dataset['Prediction'] ).sum()
TN = ( ~ dataset['price'] & ~dataset['Prediction'] ).sum()
FP = ( ~ dataset['price'] & dataset['Prediction'] ).sum()
FN = ( dataset['price'] & ~dataset['Prediction']).sum()
precision = TP/ ( TP + FP )
recall = TP / ( TP + FN)
F1 = 2 * precision * recall / ( precision + recall )
return print(f'The value of F1 is: {F1:.2f}')
# Import needed packages
import numpy as np
# Define training and testing data
percent = int(0.7*len(listings_dropna))
rows_for_training = np.random.choice(listings_dropna.index,percent, False )
training = listings_dropna.index.isin( rows_for_training )
df_train = listings_dropna[training]
df_testing = listings_dropna[~training]
# Run functions on training data
score_model(fit_model_to(df_train), df_train) # training data
accommodates 113.975227
bedrooms 113.975227
beds 113.975227
Entire Property 113.975227
Private Room 113.975227
Hotel Room 113.975227
Shared Room 113.975227
Amount 113.975227
dtype: float64
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:18: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
The proportionn of the predictions that were correct is: 0.0032091750841750843
The value of F1 is: 0.41
# Run functions on testing data
score_model(fit_model_to(df_train), df_testing)
accommodates 113.975227
bedrooms 113.975227
beds 113.975227
Entire Property 113.975227
Private Room 113.975227
Hotel Room 113.975227
Shared Room 113.975227
Amount 113.975227
dtype: float64
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:17: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:18: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
The proportionn of the predictions that were correct is: 0.00319135878237388
The value of F1 is: 0.42
Determining if certain LA neighborhoods have significantly higher prices
# Calculates every Neighborhood's Average Price
neighborhood_mean_price = listings_cleaned[['neighbourhood_cleansed','price']].groupby('neighbourhood_cleansed').mean('price').sort_values(by='price')
# Overview of data
neighborhood_mean_price
# Determine the min and max price and what LA neighborhood it is
min_price = '{:,.2f}'.format(neighborhood_mean_price['price'].min())
min_index = neighborhood_mean_price['price'].idxmin() # retrieves the index of the min value
max_price = '{:.2f}'.format(neighborhood_mean_price['price'].max())
max_index = neighborhood_mean_price['price'].idxmax() # retrieves the index of the max value
# Prints out values we calculated
(min_index,min_price,max_index, max_price)
# Import package needed
from scipy import stats
# Hypothesis Testing for Neighborhood Means
max_neighborhood = listings_cleaned[listings_cleaned['neighbourhood_cleansed'] == max_index]['price']
min_neighborhood = listings_cleaned[listings_cleaned['neighbourhood_cleansed'] == min_index]['price']
α = 0.05
t_statistics, p_value = stats.ttest_ind(min_neighborhood,max_neighborhood,equal_var=False )
reject_H0 = p_value < α
reject_H0, p_value, t_statistics
/shared-libs/python3.7/py/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3622: RuntimeWarning: Degrees of freedom <= 0 for slice
**kwargs)
/shared-libs/python3.7/py/lib/python3.7/site-packages/numpy/core/_methods.py:226: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
# Calculates Top 10 LA Neighborhoods' Average Price and sort by lowest to highest price
top_10_neighborhoods = ['Mar Vista', 'Westwood', 'Bel-Air', 'Porter Ranch', 'West Los Angeles', 'Pacific Palisades', 'Brentwood', 'Palms', 'Sawtelle', 'Century City']
top_neighborhoods = listings_cleaned[listings_cleaned['neighbourhood_cleansed'].isin(top_10_neighborhoods)]
top_neighborhood_means = top_neighborhoods[['neighbourhood_cleansed','price']].groupby('neighbourhood_cleansed').mean('price').sort_values(by='price')
# Visualize the mean neighborhood values
top_neighborhood_means
# Barplot showing mean Airbnb price per
sns.barplot(x=top_neighborhood_means.index,y=top_neighborhood_means['price'])
sns.set_style('dark')
sns.set_palette('Spectral')
plt.title("Mean Price of Top 10 Neighborhoods ")
plt.xlabel('Top 10 Neighborhoods', fontsize = 14)
plt.ylabel('Price (in US dollars)', fontsize = 14)
plt.xticks(rotation=90)
plt.show()
# Determine the min and max price for top 10 and what LA neighborhood it is
min_price_top = '{:,.2f}'.format(top_neighborhood_means['price'].min())
min_index_top = top_neighborhood_means['price'].idxmin() # retrieves the index of the min value
max_price_top = '{:.2f}'.format(top_neighborhood_means['price'].max())
max_index_top = top_neighborhood_means['price'].idxmax() # retrieves the index of the max value
# Prints out values we calculated
(min_index_top,min_price_top,max_index_top, max_price_top)
# Hypothesis Testing for Top and Bottom Neighborhood Means
top_neighborhood = top_neighborhoods[top_neighborhoods['neighbourhood_cleansed']==max_index_top]['price']
bottom_neighborhood = top_neighborhoods[top_neighborhoods['neighbourhood_cleansed']==min_index_top]['price']
α = 0.05
t_statistics, p_value = stats.ttest_ind(bottom_neighborhood,top_neighborhood,equal_var=False )
reject_H0 = p_value < α
reject_H0, p_value, t_statistics