Exam 2021 - Duplicate

import numpy as np import pandas as pd import matplotlib.pyplot as plt import warnings warnings.filterwarnings("ignore")

data_stock = pd.read_csv("../Data/Stock_fundamentals_price_2019.csv", sep = ",") data_stock.head()

A) Clean and Manipulate the data.

1.Import all the necessary packages and functions you will need to perform the exercise

import pandas as pd import numpy as np import time from sklearn.preprocessing import PolynomialFeatures from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import RandomForestClassifier from sklearn.linear_model import Ridge, Lasso, LogisticRegression from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score, classification_report from sklearn.metrics import precision_score, roc_curve from sklearn.model_selection import train_test_split, GridSearchCV import matplotlib.pyplot as plt import warnings warnings.filterwarnings("ignore")

2. Compute the percentage of NaN for each column and drop the columns that have more than 50% of NaN

NAN_list = list(data_stock.loc[:,data_stock.isnull().sum()/len(data_stock) > 0.5].columns) data_stock = data_stock.drop(NAN_list, axis = 1) data_stock

3. Create a list which contains all the names of the columns with at least one NaN value. (Hint: % bigger than zero, use command .loc()). Drop the variables (features) with more than 30% of missing values.

NAN_list_2 = list(data_stock.loc[:,data_stock.isnull().sum() > 0].columns) NAN_list_3 = list(data_stock.loc[:,data_stock.isnull().sum()/len(data_stock) > 0.3].columns) data_stock = data_stock.drop(NAN_list_3, axis = 1)

4. Fill the NaN using the median value of the column grouped by “Sector”.

data_stock_fillna_median = data_stock.fillna(data_stock.groupby('Sector').transform('median')) print("number of missing values: %d" %data_stock_fillna_median.isnull().sum().sum())

number of missing values: 0

5. Check the type of the variables, dividing them among Categorical, Dummy, and Numerical. (Hint: use : select_dtypes(include=[type to include]). Are there any dummy variables? If so, convert them.6. Compute the correlation matrix and show the top 20 variables correlated with 2019_PRICE_VAR.

data_stock_f = data_stock_fillna_median.select_dtypes(include=['float64']) data_stock_o = data_stock_fillna_median.select_dtypes(include=['object']) sector_dummies = pd.get_dummies(data_stock_o['Sector'], drop_first=True) data_stock_merge = pd.concat([data_stock_fillna_median.drop('Sector', axis=1), sector_dummies], axis=1)

6. Compute the correlation matrix and show the top 20 variables correlated with 2019_PRICE_VAR.

correlations = data_stock_merge.corr().loc["2019_PRICE_VAR"].sort_values(ascending = False).head(21)

7. Pick 10 variables between the top 20 in point 6 and plot the scatter plot between them and 2019_PRICE_VAR. N.B.: if the target (Class) in between the top 20 DO NOT PICK IT.

names_correlations = list(correlations.index[2:12]) for i in names_correlations: plt.figure() plt.scatter(data_merge[i], data_merge['2019_PRICE_VAR']) plt.title("variable "+i+" vs price") plt.grid()

8. Drop the variables that in your opinion are not needed further. You can decide to not drop any variables. N.B.: Drop '2019_PRICE_VAR' 

# Drop unnamed variable in Question 10

9. Pick 10 out of the 20 top correlated variables at point 6 and compute the nonlinear features, using degree=2 and including the interaction term, not the bias, and add them to the original data as predictors.

poly_fit = PolynomialFeatures(degree = 2, interaction_only = True, include_bias= False) data_nonlin = poly_fit.fit_transform(data_stock_merge[['dividendpaidAndCapexCoverageRatios', 'capitalExpenditureCoverageRatios',"5Y Revenue Growth (per Share)", "5Y Operating CF Growth (per Share)","5Y Net Income Growth (per Share)","PE ratio","priceEarningsRatio","5Y Shareholders Equity Growth (per Share)", "Book Value per Share", "Enterprise Value"]]) data_nonlin = pd.DataFrame(data_nonlin) data_stock_merge_2 = pd.concat([data_stock_merge.drop(['dividendpaidAndCapexCoverageRatios', 'capitalExpenditureCoverageRatios',"5Y Revenue Growth (per Share)", "5Y Operating CF Growth (per Share)","5Y Net Income Growth (per Share)","PE ratio","priceEarningsRatio","5Y Shareholders Equity Growth (per Share)", "Book Value per Share", "Enterprise Value"], axis=1), data_nonlin], axis=1) data_stock_merge_2 = data_stock_merge_2.drop(['Unnamed: 0'], axis=1)

data_stock_merge = data_stock_merge.drop(['Unnamed: 0'], axis=1)

10. Define the matrix of the predictors (X), and the target Class (y) from the dataset. (which is Class)

y = data_stock_merge_2["Class"] x = data_stock_merge_2.drop("Class", axis = 1)

B) Fit and Estimate the models using a proper cross-validation.

1. Split the X and the Y into train set and test set. TEST SIZE = 0.25.

X = data_stock_merge.drop(['Class'], axis=1) y = data_stock_merge['Class'] X_train_1, X_test_1, y_train_1, y_test_1 = train_test_split(X,y,test_size=0.25, random_state=8, shuffle=True)

2. Fit the logistic regression using the X without the nonlinear feature computed at point 9.

log = LogisticRegression(solver='liblinear', random_state=8) model_1 = log.fit(X_train_1,y_train_1) B_0 = model_1.intercept_ print(' Intercept: ', B_0) coef_1 = model_1.coef_ y_pred_1_train = model_1.predict(X_train_1) y_pred_1_train p_pred_1_prob_train = model_1.predict_proba(X_train_1)

 Intercept:     [2.23228142e-17]

X_train_1

3. Evaluate the logistic regression using: accuracy, precision, confusion matrix.

conf_mat_train = confusion_matrix(y_train_1,y_pred_1_train) conf_mat_df_train = pd.DataFrame(conf_mat_train, index = ['0', '1'], columns=['0', '1']) print(conf_mat_df_train) print('') prec_train = precision_score(y_train_1,y_pred_1_train) print(' Precision train: ', prec_train) accur_train = accuracy_score(y_train_1,y_pred_1_train) print(' Accuracy train: ', accur_train) sens_train = conf_mat_df_train.iloc[0,0]/(conf_mat_df_train.iloc[0,0]+conf_mat_df_train.iloc[0,1]) print(' Sensitivity train: ', sens_train) spec_train = conf_mat_df_train.iloc[1,1]/(conf_mat_df_train.iloc[1,0]+conf_mat_df_train.iloc[1,1]) print(' Specificity train: ', spec_train) auc_score = roc_auc_score(y_train_1, p_pred_1_prob_train[:,1]) print(' Roc Auc train: ', auc_score)

     0     1
0  136   875
1  141  2142

 Precision train:    0.7099767981438515
 Accuracy train:     0.6915604128718883
 Sensitivity train:  0.13452027695351138
 Specificity train:  0.938239159001314
 Roc Auc train:      0.731856282599682

4. Print the classification report.

report_train = classification_report(y_train_1,y_pred_1_train) print(report_train)

              precision    recall  f1-score   support

           0       0.49      0.13      0.21      1011
           1       0.71      0.94      0.81      2283

    accuracy                           0.69      3294
   macro avg       0.60      0.54      0.51      3294
weighted avg       0.64      0.69      0.63      3294

5. What can you say about the overall performance of the model? Is it good? Is it better in classifying 1 or 0? Why? How is the sensitivity with respect to 0? and 1? What does this mean?

6. Fit the logistic regression using all the predictors, including also the non-linear term.

X = data_stock_merge_2.drop(['Class'], axis=1) y = data_stock_merge_2['Class'] X_train_1, X_test_1, y_train_1, y_test_1 = train_test_split(X,y,test_size=0.25, random_state=8, shuffle=True) log = LogisticRegression(solver='liblinear', random_state=8) model_1 = log.fit(X_train_1,y_train_1) B_0 = model_1.intercept_ print(' Intercept: ', B_0) coef_1 = model_1.coef_ y_pred_1_train = model_1.predict(X_train_1) y_pred_1_train p_pred_1_prob_train = model_1.predict_proba(X_train_1)

 Intercept:     [7.318248e-18]

7. Evaluate the logistic regression using: accuracy, precision, confusion matrix.

conf_mat_train = confusion_matrix(y_train_1,y_pred_1_train) conf_mat_df_train = pd.DataFrame(conf_mat_train, index = ['0', '1'], columns=['0', '1']) print(conf_mat_df_train) print('') prec_train = precision_score(y_train_1,y_pred_1_train) print(' Precision train: ', prec_train) accur_train = accuracy_score(y_train_1,y_pred_1_train) print(' Accuracy train: ', accur_train) sens_train = conf_mat_df_train.iloc[0,0]/(conf_mat_df_train.iloc[0,0]+conf_mat_df_train.iloc[0,1]) print(' Sensitivity train: ', sens_train) spec_train = conf_mat_df_train.iloc[1,1]/(conf_mat_df_train.iloc[1,0]+conf_mat_df_train.iloc[1,1]) print(' Specificity train: ', spec_train) auc_score = roc_auc_score(y_train_1, p_pred_1_prob_train[:,1]) print(' Roc Auc train: ', auc_score)

     0     1
0  132   879
1  142  2141

 Precision train:    0.7089403973509933
 Accuracy train:     0.6900425015179114
 Sensitivity train:  0.13056379821958458
 Specificity train:  0.9378011388523873
 Roc Auc train:      0.7300899046103896

8. Print the classification report.

report_train = classification_report(y_train_1,y_pred_1_train) print(report_train)

              precision    recall  f1-score   support

           0       0.48      0.13      0.21      1011
           1       0.71      0.94      0.81      2283

    accuracy                           0.69      3294
   macro avg       0.60      0.53      0.51      3294
weighted avg       0.64      0.69      0.62      3294

9. What can you say about the overall performance of the model? Has it improved from the one of the logist regression above? If yes, why?

10. Using the entire dataset at point 10, fit the penalized logistic regression using Ridge penalization. In order to calibrate the optimal α\alphaα use the GridSearchCV function. Use at least 5 folds for the cross-validation (cv=5) and use at least 50 values for alpha. As scoring use the neg_log_loss. Which is the optimal value of α\alphaα. HINT: remember to 'scale' the data.

X_2 = data_w_interactions.drop(['Class', 'operatingProfitMargin', '2019_PRICE_VAR'], axis = 1) y_ridge = data_w_interactions[['Class']] X_train_2, X_test_2, y_train_2, y_test_2 = train_test_split(X_2, y_2, test_size = 0.25, random_state=8, shuffle=True) X_train_std_2 = (X_train_ridge - X_train_ridge.mean())/X_train_ridge.std() X_test_std_ridge = (X_test_ridge - X_train_ridge.mean())/X_train_ridge.std() LogReg_ridge = LogisticRegression(penalty='l2', solver='liblinear', random_state=3) params = {'C': np.linspace(0,5,50)} LogReg_ridge_gscv = GridSearchCV(estimator = LogReg_ridge, param_grid = params, cv = 5, scoring = 'neg_log_loss', return_train_score= True, refit = True)

11. Plot the cross-validation test neg_los_loss value verus the value of alpha. How is the graph? Did you find the optimal α\alphaα value? If no, what could be the issue?

12. Evaluate the Ridge penalized logistic regression using: accuracy, precision, confusion matrix.

13. Using the entire dataset at point 10, fit the penalized logistic regression using LASSO penalization. In order to calibrate the optimal α\alphaα use the GridSearchCV function. Use at least 5 folds for the cross-validation (cv=5) and use at least 50 values for alpha. As scoring use the neg_log_loss. Which is the optimal value of α\alphaα. 

14. Plot the cross validation test neg_los_loss value verus the value of alpha. How is the graph? Did you find the optimal α\alphaα value? If no, what could be the issue?

15. Evaluate the Lasso penalized logistic regression using: accuracy, precision, confusion matrix.

16. Has the shrinkage (Ridge or Lasso) improved the overall performance with respect to the Logisitc regression? Why?

17. Using the entire dataset at point 10, fit the Decision Tree Classifier. In order to calibrate the oprimal value for: max_depth and min_samples_split, use the GridSearchCV function. Use at least 5 folds for the cross-validation (cv=5) and use at least 5 values for the max_depth and 5 for the min_samples_split. As scoring use the neg_log_loss. Which is the optimal value of max_depth and min_samples_split? 

18. Re-fit the best model and plot the feature importance bar plot. Plot only the top 20 variables.

19. Evaluate the Decision Tree Clssifier using: accuracy, precision, confusion matrix.

20. Using the entire dataset at point 10, fit the Random Forest Classifier. In order to calibrate the oprimal value for: max_depth and min_samples_split, use the GridSearchCV function. Use at least 5 folds for the cross-validation (cv=5) and use at least 5 values for n_estimators, 5 values for the max_depth and 5 for the min_samples_split. As scoring use the neg_log_loss. Which is the optimal value of n_estimators, max_depth, and min_samples_split? 

21. Re-fit the best model and plot the feature importance bar plot. Plot only the top 20 variables.

22. Evaluate the Random Forest Classifier using: accuracy, precision, confusion matrix.

23. The top 20 features have changed between Point 18 and 21? If yes, why and how?

24. Compare the performance of the Decision Tree Classifier and the Random Forest Classifier. Which one has the best precision and accuracy? Why?

25. Which is the best model overall in terms of performances? Why? This could be explained by the data structure or by the model structure? Is there room to improve the performance? If so, how?

C) BEST MODEL MONEY PERFORMANCE.

1. Using the best model determined in point 25, with its optimal hyperparameters, do the following:Use the index of y_test to select the corresponding: 2019_PRICE_VAR.Multiply the selected 2019_PRICE_VAR by y_pred, which is a vector of 1 and 0Sum the resulting vector which will be your portfolio price variation.

2. Is the portfolio price variation computed using the test sample positive? If so, what does it mean?

.css-9i38tn{letter-spacing:inherit;-webkit-text-decoration:none;text-decoration:none;}

A) Clean and Manipulate the data.

B) Fit and Estimate the models using a proper cross-validation.

C) BEST MODEL MONEY PERFORMANCE.