Credit Card Fraud Detection using Gradient Boosting Trees

Import libraries

import sys !{sys.executable} -m pip install imblearn | grep -v 'Requirement already satisfied' !{sys.executable} -m pip install ipywidgets | grep -v 'Requirement already satisfied'

Collecting imblearn
  Downloading imblearn-0.0-py2.py3-none-any.whl (1.9 kB)
Collecting imbalanced-learn
  Downloading imbalanced_learn-0.10.1-py3-none-any.whl (226 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 226.0/226.0 KB 35.3 MB/s eta 0:00:00
Installing collected packages: imbalanced-learn, imblearn
Successfully installed imbalanced-learn-0.10.1 imblearn-0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Collecting ipywidgets
  Downloading ipywidgets-8.0.4-py3-none-any.whl (137 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.8/137.8 KB 23.8 MB/s eta 0:00:00
Collecting jupyterlab-widgets~=3.0
  Downloading jupyterlab_widgets-3.0.5-py3-none-any.whl (384 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 384.3/384.3 KB 7.0 MB/s eta 0:00:00
Collecting widgetsnbextension~=4.0
  Downloading widgetsnbextension-4.0.5-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 112.9 MB/s eta 0:00:00
Installing collected packages: widgetsnbextension, jupyterlab-widgets, ipywidgets
Successfully installed ipywidgets-8.0.4 jupyterlab-widgets-3.0.5 widgetsnbextension-4.0.5
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

from imblearn.over_sampling import RandomOverSampler

import ipywidgets as widgets from ipywidgets import interact

import matplotlib.pyplot as plt import pandas as pd import seaborn as sns from sklearn.impute import SimpleImputer from sklearn.model_selection import train_test_split from sklearn.pipeline import make_pipeline from sklearn.ensemble import GradientBoostingClassifier from sklearn.metrics import ( ConfusionMatrixDisplay, classification_report, confusion_matrix, ) from sklearn.model_selection import GridSearchCV, train_test_split

Read the data

df = pd.read_csv("/work/creditcard.csv")

The data was obtained from Kaggle.

https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud

This dataset presents transactions that occurred in two days. The column Class take the values 0 an 1, 1 if there is a fraud in the transaction.

Explore

We are going to explore the dataset. First we are going to see the head of it.

df.head()

Timefloat64

V1float64

0.0

-1.3598071336738

0.0

1.19185711131486

1.0

-1.35835406159823

1.0

-0.966271711572087

2.0

-1.15823309349523

df.shape

df.isnull().sum().sum()

We can see that there is not null values.

Now we are going to plot the Class Balance.

# Plot class balance df["Class"].value_counts(normalize=True).plot( kind="bar", xlabel="Fraud", ylabel="Frequency", title="Class Balance" );

We have an imbalanced dataset. Our majority class is far bigger than our minority class.

Split

We are going to split the data frame in X and y and then we are going to use train test plit to obtain X train, X test, y train and y test.

target = "Class" X = df.drop(columns=target) y = df[target] print("X shape:", X.shape) print("y shape:", y.shape)

X shape: (284807, 30)
y shape: (284807,)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) print("X_train shape:", X_train.shape) print("y_train shape:", y_train.shape) print("X_test shape:", X_test.shape) print("y_test shape:", y_test.shape)

X_train shape: (227845, 30)
y_train shape: (227845,)
X_test shape: (56962, 30)
y_test shape: (56962,)

Resample

We see that the data set is imbalanced.

y_train.value_counts()

We are going to cretae a new feature matrix X_train_over and target vector y_train_over by performing random over-sampling on our training data. We choose over sampling because we saw in the previous project that under sampling was not good.

over_sampler = RandomOverSampler(random_state=42) X_train_over, y_train_over = over_sampler.fit_resample(X_train, y_train) print(X_train_over.shape) X_train_over.head()

(454902, 30)

Timefloat64

V1float64

143352.0

1.95504092199146

117173.0

-0.400975238728654

149565.0

0.0725090163689562

93670.0

-0.535045380949255

82655.0

-4.02693795043132

y_train_over.value_counts(normalize=True)

Model

Baseline

We calculate the baseline accuracy score for our model.

acc_baseline = y_train.value_counts(normalize=True).max() print("Baseline Accuracy:", round(acc_baseline, 4))

Baseline Accuracy: 0.9983

Iterate

We are going to create a model named clf, it is going to contain Gradient Boosting Classifier

clf = make_pipeline(GradientBoostingClassifier(random_state=42)) print(clf)

Pipeline(steps=[('gradientboostingclassifier',
                 GradientBoostingClassifier(random_state=42))])

Now we create a dictionary with the range of hyperparameters that we are going to evaluate for our classifier.

params = { "gradientboostingclassifier__max_depth":range(2,5), "gradientboostingclassifier__n_estimators":range(20,31,5) } params

We create a GridSearchCV to which we called model. This is going to include clf and hyperparameter grid.

model = GridSearchCV( clf, param_grid = params, cv = 5, n_jobs = -1, verbose = 1 ) model

We fit model to the over-sampled training data.

model.fit(X_train_over, y_train_over)

Fitting 5 folds for each of 9 candidates, totalling 45 fits

Now we extract the cross-validation results from the model

results = pd.DataFrame(model.cv_results_) results.sort_values("rank_test_score").head(10)

mean_fit_timefloat64

40.29000315666199 - 114.17910232543946

std_fit_timefloat64

0.17956618130221969 - 4.266099356385528

114.17910232543946

4.266099356385528

96.36223621368408

0.36948240608282357

76.98166213035583

0.6307917268536979

90.68267741203309

0.9608229506759278

74.09340267181396

0.39072676349905877

60.154942464828494

0.3031443893893782

62.33220171928406

0.5278362134150776

51.12425374984741

0.6441260278476304

40.29000315666199

0.17956618130221969

Now we extract the best hyperparameters from model.

# Extract best hyperparameters model.best_params_

Evaluate

We are going to evaluate the model.

Calculate the training and test accuracy scores for model.

acc_train = model.score(X_train, y_train) acc_test = model.score(X_test, y_test) print("Training Accuracy:", round(acc_train, 4)) print("Test Accuracy:", round(acc_test, 4))

Training Accuracy: 0.9894
Test Accuracy: 0.9891

Now we make the confusion Matrix. First we count how many observation in y_test belong to positive and negative class.

y_test.value_counts()

Now we plot hte confusion matrix.

ConfusionMatrixDisplay.from_estimator(model, X_test, y_test);

We can see that there are more true positives than when we used the Random Forest Classifier. In this case of projects, we need to understand what we need. If we want to reduce the false positives or the false negatives. Of this depends on which algorithm we are going to use.

Now we are going to see the precision and the recall of this model, this are going to be more important to evaluate it than the accuracy score.

Now we print the classification report for the model, using the test set.

# Print classification report print(classification_report(y_test, model.predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      0.99      0.99     56864
           1       0.13      0.92      0.22        98

    accuracy                           0.99     56962
   macro avg       0.56      0.95      0.61     56962
weighted avg       1.00      0.99      0.99     56962

We can see that the recall is higher than the precision for the positive class. This is because in this model the False Negatives are lower, and the True Positives are higher, for this reason recall is higher. In other hand, the False Negatives are higher, that is the reason why the precision is lower.

Finally, we are going to make a function to make predictions in which we can change the threshold. This is a parameter that affects the recall and precision.

def make_cnf_matrix(threshold): y_pred_proba = model.predict_proba(X_test)[:,-1] y_pred = y_pred_proba>threshold conf_matrix = confusion_matrix(y_test, y_pred) tn, fp, fn, tp = conf_matrix.ravel() #print(f"Profit:€{tp*100_000_000}") ConfusionMatrixDisplay.from_predictions(y_test, y_pred, colorbar=False) thresh_widget = widgets.FloatSlider(min=0, max=1, value=0.5, step=0.05) interact(make_cnf_matrix, threshold=thresh_widget);

If the threshold is high, the recall is low and the precision high. If the threshold is low, the recall is high and the precision is low.

The value we choose of threshold depends on which one we want higher, recall or precision. This is a decision we are going to take depending on for what we are going to use the results we obtain.

Communicate

We obtain the features and importances of our model and plot them.

# Get feature names from training data features = X_train_over.columns # Extract importances from model importances = model.best_estimator_.named_steps[ "gradientboostingclassifier" ].feature_importances_ # Create a series with feature names and importances feat_imp = pd.Series(importances, index=features).sort_values() # Plot 10 most important features feat_imp.tail(10).plot(kind="barh") plt.xlabel("Gini Importance") plt.ylabel("Feature") plt.title("Feature Importance");

We can see that the most importan feature is V14.