Credit card fraud

Experimenting with SVM vs. LightGBM

import lightgbm as lgb from lightgbm import LGBMClassifier import numpy as np import os import pandas as pd from sklearn.svm import SVC from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score, classification_report, roc_auc_score, roc_curve, confusion_matrix, ConfusionMatrixDisplay from sklearn.preprocessing import StandardScaler from sklearn.utils import resample import matplotlib.pyplot as plt import plotly.express as px

Introduction.

This is a highly imbalanced dataset which will benefit from a good dose of downsampling; there are only around 500 fraudulent features, so the remaining non-fraudulent data will be trimmed-down from a size of 285,000 to a satisfactory size for training & validation, and the remaining portion of the data will be saved separately for testing.

In the initial instance, this was a project to outline the lasting clout of the SVM. As one of my favourite systems, it still holds its weight against the majority of other models today for classification tasks, with one-class SVM being as reliable a system for unsupervised anomaly detection tasks...

However (there is always one), the flipside of this project is to outline the power of gradient boosting algorithms... mostly because I cop flak for using LightGBM for practically every tabular data project I tackle. Lately I have seen some newbies on a certain professional networking website advocating for SVM et. al., so I thought this would be a good project to outline why SVM is still a fantastic choice for jobs like this, and to also outline the prowess of gradient boosting algos in comparison. Bear in mind that SVM here has undergone several rounds of gridSearch as well as randomizedSearch, being fine-tuned using the resulting best parameters of those several rounds. LightGBM returned comparably better scores across the board with no parameter tuning whatsoever, although I wanted to use more test data at the end of the project so it underwent a threefold cross-validation round. For this data, the results were better with LightGBM than they were using a PyTorch graph NN, autoencoders and isolation forests. I haven't visualised the data pre-model because there was enough visible information re: model choice using df.head().

df = pd.read_csv('creditcard.csv') df.head()

df["Class"].value_counts()

First is to separate the fraudulent variables, then save a handful of fraudulent variables aside for testing. The remaining fraudulent variables will be used for validation and the non-fraudulent variables will be downsampled and combined with the validation fraudulent ones for the train_test_split:

data_class_1 = df[df['Class'] == 1] class_1_test_samples = data_class_1.sample(n=50, random_state=42) data_class_1_training = data_class_1.drop(class_1_test_samples.index) class_0_sample = df[df['Class'] == 0].sample(n=3000, random_state=42) df_training = pd.concat([class_0_sample, data_class_1_training]).sample(frac=1, random_state=42) # Shuffle df_test_final = pd.concat([class_1_test_samples, class_0_sample.sample(n=50, random_state=42)]) # Add 50 '0's for balance training_output_path = 'model_training_data.csv' testing_output_path = 'model_final_testing_data.csv' df_training.to_csv(training_output_path, index=False) df_test_final.to_csv(testing_output_path, index=False) training_output_path, testing_output_path

data = pd.read_csv('model_training_data.csv') test_data = pd.read_csv('model_final_testing_data.csv') data.head()

And 100 samples have been set aside for testing:

len(test_data)

Zero null values in the data:

data.isna().any()

Scaling the features:

scaler = StandardScaler() X = data.drop(columns=['Class']) y = data['Class'] X_scaled = pd.DataFrame(scaler.fit_transform(X), columns=X.columns) X_scaled.head()

SVC and LightGBM models.

X = data.drop('Class', axis=1) y = data['Class']

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42, stratify=y, shuffle=True)

X_train.shape, X_test.shape

svm_model = SVC(class_weight='balanced', kernel='poly', degree=3, random_state=42, C=10, gamma='auto', probability=True) svm_model.fit(X_train, y_train)

y_pred = svm_model.predict(X_test)

accuracy = accuracy_score(y_test, y_pred) print("Accuracy:", accuracy)

SVC model vs. the test data .

100 samples from the original dataframe as test data, saved earlier:

Test set SVM metrics:

LightGBM trained, then tested using the test dataset.

test_lgb_accuracy, test_lgb_classification_report = None, None lgb_model = LGBMClassifier( random_state=42, class_weight="balanced", colsample_bytree=1, learning_rate=0.1, max_depth=10, n_estimators=200, num_leaves=70, subsample=0.8 ) lgb_model.fit(X_train, y_train) test_y_pred_lgb = lgb_model.predict(test_X) test_lgb_accuracy = accuracy_score(test_y, test_y_pred_lgb) test_lgb_accuracy

LightGBM test metrics:

Difference between SVC and LGBM performance, ROC AUC (validation data).

Fraudulent variables prepared for investigation.

fraud_svc = pd.read_csv('fraud_svc.csv') fraud_lgbm = pd.read_csv('fraud_lgbm.csv')

SVC:

LightGBM:

False negatives from each model for investigation.

false_negatives_svc = pd.read_csv('false_negatives_svc.csv') false_negatives_lgbm = pd.read_csv('false_negatives_lgbm.csv')

SVC:

LightGBM:

Comparison of averages between fraud variables and false negatives.

false_negatives_lgbm_time_series = false_negatives_lgbm.groupby(false_negatives_lgbm['Time'] // 5000).size() fraud_lgbm_time_series = fraud_lgbm.groupby(fraud_lgbm['Time'] // 5000).size() false_negatives_lgbm_time_series, fraud_lgbm_time_series

The total amount of money lost due to the false negatives:

money_lost_false_negatives = false_negatives_lgbm['Amount'].sum() money_lost_false_negatives

The sum of money rescued in the true positives:

Percentage of capital misrepresented per 1048 variables through this model:

Both models using 50% of the original dataframe.

Going big with 142,158 variables to see how well things have generalised:

original_data = pd.read_csv('creditcard.csv') original_data = original_data.sample(frac=0.5, random_state=42) used_class_1_indices = pd.concat( [ data_class_1_training.index.to_series(), class_1_test_samples.index.to_series() ] ) remaining_data = original_data.drop(index=used_class_1_indices, errors='ignore') remaining_data_class_1 = remaining_data[remaining_data['Class'] == 1] remaining_data_class_1.shape[0], remaining_data.shape[0]

Augmenting 20 rows of "fraudulent" data as all of the original fraudulent variables were used in training and validation:

original_fraudulent_data = original_data[original_data['Class'] == 1] augmented_fraudulent_data = resample( original_fraudulent_data, replace=False, # No exact duplicates n_samples=20, random_state=42 ).copy() numeric_columns = augmented_fraudulent_data.select_dtypes(include=['float64', 'int64']).columns for column in numeric_columns: augmented_fraudulent_data[column] += np.random.normal(0, 0.01, size=augmented_fraudulent_data[column].shape) augmented_fraudulent_data

Inserting them into the test dataframe:

remaining_data_augmented = pd.concat([remaining_data, augmented_fraudulent_data], ignore_index=True) remaining_data_augmented['Class'].value_counts(), remaining_data_augmented.shape

Converting to a workable format:

X_remaining = remaining_data_augmented.drop(['Class'], axis=1) y_remaining = remaining_data_augmented['Class'] y_remaining_binarized = (y_remaining > 0.5).astype(int) X_remaining_scaled = pd.DataFrame(scaler.transform(X_remaining), columns=X_remaining.columns) remaining_data_pred = lgb_model.predict(X_remaining_scaled) remaining_data_accuracy = accuracy_score(y_remaining_binarized, remaining_data_pred) remaining_data_accuracy

The SVM model on the augmented "real world" test dataset has returned many more false positives compared to the LightGBM model, as well ass one false negative:

The LGBM model, although trained on a small amount of data, has returned all 20 fraudulent examples and allowed for zero false negatives. There are however 368 false positives which won't cause the business to lose money as false negatives would, although this will cost money in wages due to the time required for an employee to verify the data.

Getting those numbers down as much as possible will be a work in progress from now so changes will be noticeable in this project (training / validation data size etc.), but this outlines the power of LightGBM in instances where strong models such as SVM still occasionally 'miss', especially as far as false negatives go.

.css-15w88e5{color:var(--chakra-colors-fg-neutral-primary);font-weight:inherit;letter-spacing:-0.09px;}Experimenting with SVM vs. LightGBM