IEOR142_Midterm

Data Analysis Exercise #1

!pip install statsmodels==0.13.0

import pandas as pd import numpy as np import statsmodels.api as sm

concrete_train = pd.read_csv('concrete_train.csv') concrete_test = pd.read_csv('concrete_test.csv')

concrete_train.head() #note: all variables are numerical.

#check if there is any missing values concrete_train.isnull().any()

#1--build a model with all variables import statsmodels.formula.api as smf ols = smf.ols(formula='Strength ~ Cement + Blast_Furnace_Slag + Fly_Ash + Water + Superplasticizer + Coarse_Aggregate + Fine_Aggregate + Age + Water_Cement_Ratio', data=concrete_train) model1 = ols.fit() print(model1.summary())

R^2 with all variables is 0.603. The highest p-value in this model is 0.372 from Water_Cement_Ratio.

#calculate VIF for all variables #code from HW1 from statsmodels.stats.outliers_influence import variance_inflation_factor def VIF(df, columns): values = sm.add_constant(df[columns]).values num_columns = len(columns)+1 vif = [variance_inflation_factor(values, i) for i in range(num_columns)] return pd.Series(vif[1:], index=columns) VIF(concrete_train, concrete_train.columns)

We would want to remove variables with very high VIFs; in this case, Cement has the highest value with around 13.9.

#2 -- remove 'Cement' to see changes in quality ols2 = smf.ols(formula='Strength ~ Blast_Furnace_Slag + Fly_Ash + Water + Superplasticizer + Coarse_Aggregate + Fine_Aggregate + Age + Water_Cement_Ratio', data=concrete_train) model2 = ols2.fit() print(model2.summary()) print(VIF(concrete_train, ['Blast_Furnace_Slag','Fly_Ash','Water','Superplasticizer','Coarse_Aggregate','Fine_Aggregate','Age','Water_Cement_Ratio']))

The VIFs have gone down to below 5 for all variables except for Water. R^2 score has gone down to 0.560, which may mean that removing Cement might not be a good idea. P-value of Fly_Ash has become extremely high. Let's check if removing other variables would lower the VIFs instead.

#3--get back 'Cement', remove 'Water' ols3 = smf.ols(formula='Strength ~ Cement + Blast_Furnace_Slag + Fly_Ash + Superplasticizer + Coarse_Aggregate + Fine_Aggregate + Age + Water_Cement_Ratio', data=concrete_train) model3 = ols3.fit() print(model3.summary()) print(VIF(concrete_train, ['Cement','Blast_Furnace_Slag','Fly_Ash','Superplasticizer','Coarse_Aggregate','Fine_Aggregate','Age','Water_Cement_Ratio']))

The R^2 score (0.60) has gone up compared to model #2 and is lower than model #1 by 0.003. The p-values seem to be the lowest overall so far. However, VIF score is again extremely high for Cement. Would choose model 3 over model 1 and 2. What if we take out both Cement and Water?

#4--remove both 'Cement' and 'Water' ols4 = smf.ols(formula='Strength ~ Blast_Furnace_Slag + Fly_Ash + Superplasticizer + Coarse_Aggregate + Fine_Aggregate + Age + Water_Cement_Ratio', data=concrete_train) model4 = ols4.fit() print(model4.summary()) print(VIF(concrete_train, ['Blast_Furnace_Slag','Fly_Ash','Superplasticizer','Coarse_Aggregate','Fine_Aggregate','Age','Water_Cement_Ratio']))

The R^2 score of this model is the lowest. Considering the p-value for model #3 is much more significant than #4, we should probably keep 'Cement', therefore, model 3 is the best so far. Going back to the result of #1 (all variables present), what if we remove Blast_Furnace_Slag as it has the second to highest VIF?

#5 -- all variables except 'Blast_Furnace_Slag' ols5 = smf.ols(formula='Strength ~ Cement + Water + Fly_Ash + Superplasticizer + Coarse_Aggregate + Fine_Aggregate + Age + Water_Cement_Ratio', data=concrete_train) model5 = ols5.fit() print(model5.summary()) print(VIF(concrete_train, ['Cement','Water','Fly_Ash','Superplasticizer','Coarse_Aggregate','Fine_Aggregate','Age','Water_Cement_Ratio']))

R^2 score is 0.567, and the p-value of Water_Cement_Ratio is quite high at 0.791. None of the variables have the VIF higher than 10 like model 3 does. However, Cement and Water_Cement_Ratio still have moderately high VIF. Last model--try removing Water_Cement_Ratio as it may be linearly dependent with Water and Cement columns.

#6 -- remove Water_Cement_Ratio ols6 = smf.ols(formula='Strength ~ Cement + Blast_Furnace_Slag + Fly_Ash + Water + Superplasticizer + Coarse_Aggregate + Fine_Aggregate + Age', data=concrete_train) model6 = ols6.fit() print(model6.summary()) print(VIF(concrete_train, ['Cement','Blast_Furnace_Slag','Water','Fly_Ash','Superplasticizer','Coarse_Aggregate','Fine_Aggregate','Age']))

R^2 score is 0.602, lower than model 1 but higher than model 3. P-values all seem to be fairly low and significant, and though VIF values are moderately high for majority of the variables, Cement is no longer higher than 10 in this model. With that in mind, I will set model 6 (all variables except Water_Cement_Ratio) to be my final linear regression model. According to the summary, Superplasticizer, Cement, and Age have the highest direct correlation with strength of the cement. On the other hand, Water has a reverse correlation with the dependent variable, meaning that the higher the amount of water there is, the lower the strength of the concrete is.

#test the performance on the test data set #create OSR^2 function from Homework 1 def OSR2(model, df_train, df_test, dependent_var): y_test = df_test[dependent_var] y_pred = model.predict(df_test) SSE = np.sum((y_test - y_pred)**2) SST = np.sum((y_test - np.mean(df_train[dependent_var]))**2) return 1 - SSE/SST

OSR2(model6, concrete_train, concrete_test, 'Strength')

The OSR^2 score on the test set is 0.633, higher than R^2 which is a good indicator that the model is less likely to be overfitting.

.css-15w88e5{color:var(--chakra-colors-fg-neutral-primary);font-weight:inherit;letter-spacing:-0.09px;}Data Analysis Exercise #1

Data Analysis Exercise #1