IB031 - YouTube comments
autoři: Gendiar, Košvanec, Tsykynovska
Načtení datasetu
Načteme všechné soubory do jednoho DataFrame a vypíšeme základní info:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
sns.set() # make plots nicer
df1 = pd.read_csv ('data/Youtube01-Psy.csv', parse_dates=["DATE"])
df2 = pd.read_csv ('data/Youtube02-KatyPerry.csv', parse_dates=["DATE"])
df3 = pd.read_csv ('data/Youtube03-LMFAO.csv', parse_dates=["DATE"])
df4 = pd.read_csv ('data/Youtube04-Eminem.csv', parse_dates=["DATE"])
df5 = pd.read_csv ('data/Youtube05-Shakira.csv', parse_dates=["DATE"])
df = pd.concat([df1, df2, df3, df4, df5])
df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1956 entries, 0 to 369
Data columns (total 5 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 COMMENT_ID 1956 non-null object
1 AUTHOR 1956 non-null object
2 DATE 1711 non-null datetime64[ns]
3 CONTENT 1956 non-null object
4 CLASS 1956 non-null int64
dtypes: datetime64[ns](1), int64(1), object(3)
memory usage: 91.7+ KB
Explorační analýza
Prozkoumáme dataset. Podíváme se, kolik v něm je spamů a hamů, jaký je jejich podíl, jaká jsou nejpouživánějsí slova pro spam a ham, kolik slov je ve zprávách a v jaké dny týdnu zprávy jsou nejčastěji přidávané.
Tady vidíme, že data jsou rozdělené na ham a spam přibližně stejně.
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
sns.set() # make plots nicer
labels = 'Spam', 'Ham'
plt.pie(df['CLASS'].value_counts(), labels=labels, autopct='%1.1f%%', radius=2.0)
plt.show()
Mezi nejpoužívanější slova ke spamu patří "check", "please", "subscribe", "channel", k hamu "love", "song", "like".
from nltk.corpus import stopwords
import nltk
nltk.download("stopwords")
def remove_stop_words(comments):
return [word for word in ' '.join(comments).lower().split()
if word not in (stopwords.words('english') + [",", ".", "", "2"])]
spam_comments = df[df["CLASS"] == 1]["CONTENT"]
ham_comments = df[df["CLASS"] == 0]["CONTENT"]
spam_list = remove_stop_words(spam_comments)
ham_list = remove_stop_words(ham_comments)
print("Spam - 10 most common used words")
print(pd.Series(spam_list).value_counts()[:10])
print("Ham - 10 most common used words")
print(pd.Series(ham_list).value_counts()[:10])
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
Spam - 10 most common used words
check 506
video 208
please 201
subscribe 193
like 147
channel 124
youtube: 120
new 119
guys 94
music 92
dtype: int64
Ham - 10 most common used words
song 146
love 136
like 78
video 65
katy 51
best 49
views 44
billion 43
good 35
people 32
dtype: int64
Chceme vidět, jaká je závislost mezi počtem slov ve zprávě a její příslušností ke spamu/hamu.
df["WORD_COUNT"] = df["CONTENT"].apply(lambda x: len(x.split()))
df
Zajímavé je, že velká část spamových zpráv má 6 slov.
dfspam_wordcount = df[df["CLASS"] == 1]["WORD_COUNT"].value_counts()
dfham_wordcount = df[df["CLASS"] == 0]["WORD_COUNT"].value_counts()
dfcombined = pd.DataFrame({'spam': dfspam_wordcount, 'ham': dfham_wordcount})
dfcombined.plot.bar(figsize=(20, 5), stacked=True, xlabel="Word count")
Každý den bylo přidávané skoro stejné množství hamů a spamů, proto jsme rozhodli nepoužívat tento sloupec.
df_with_date = pd.concat([df1, df2, df3, df5])
df_with_date["WEEKDAY"] = df_with_date.DATE.dt.weekday
dfspam_weekday = df_with_date[df_with_date["CLASS"] == 1]["WEEKDAY"].value_counts()
dfham_weekday = df_with_date[df_with_date["CLASS"] == 0]["WEEKDAY"].value_counts()
dfcombined = pd.DataFrame({'spam': dfspam_weekday, 'ham': dfham_weekday})
dfcombined.plot.bar(figsize=(20, 5), stacked=True, xlabel="Weekday")
Předzpracování
from sklearn.feature_extraction.text import CountVectorizer
comments = df[["CONTENT", "CLASS"]]
count_vectorizer = CountVectorizer().fit(comments.CONTENT)
bag_of_words = count_vectorizer.transform(comments.CONTENT)
print("Number of columns:", bag_of_words.shape[1])
Number of columns: 4454
from sklearn.model_selection import train_test_split
comments_X, comments_y = bag_of_words, df.CLASS
comments_train_X, comments_test_X, comments_train_y, comments_test_y = train_test_split(
comments_X, comments_y, test_size=0.2, random_state=42
)
comments_train_text, comments_test_text, _, _ = train_test_split(
df["CONTENT"], comments_y, test_size=0.2, random_state=42
)
Modely
Baseline model - Naive Bayes
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import f1_score
pipeline_multinomial_NB = make_pipeline(
MultinomialNB()
)
pipeline_multinomial_NB.fit(comments_train_X, comments_train_y)
print("Baseline model - Naive Bayes accuracy", round(pipeline_multinomial_NB.score(comments_test_X, comments_test_y) * 100.0, 3), "%")
print("F1 score", f1_score(pipeline_multinomial_NB.predict(comments_test_X), comments_test_y, average=None))
Baseline model - Naive Bayes accuracy 92.092 %
F1 score [0.91267606 0.92773893]
from sklearn import metrics
sns.set_style("whitegrid", {"axes.grid": False})
def confusion_matrix(pipeline):
disp = metrics.plot_confusion_matrix(pipeline, comments_test_X, comments_test_y, display_labels=['ham', 'spam'])
disp.figure_.suptitle("Confusion Matrix")
confusion_matrix(pipeline_multinomial_NB)
from sklearn.model_selection import cross_val_score
cross_val_scores_nb = cross_val_score(pipeline_multinomial_NB, comments_X, comments_y, cv=50)
print("50-fold cross validation average score for Naive Bayes model: ", round(np.mean(cross_val_scores_nb) * 100.0, 3), "%")
50-fold cross validation average score for Naive Bayes model: 91.973 %
Decision trees model
Tato metoda se používá pro klasifikaci a regrese. Vybrali jsme DecisionTreeClassifier, protože potřebujeme rozdělit data na několik diskretních tříd. Celkově pro Decision Trees platí, že cím hlubší je strom, tím složitější jsou rozhodovácí pravidla a tím napasovaný je model. Ale z toho vyplývá nedostatek této metody, t.z. "overfitting".
Pro hledání parametrů, které by dali největší správnost, používáme GridSearchCV.
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
parameters = {
"random_state" : [42],
"max_depth": [i for i in range(1, 25, 2)],
"min_samples_leaf": [i for i in range(1, 10, 2)],
"criterion": ["gini"],
"ccp_alpha": [0.0, 0.001, 0.01, 0.1, 0.3],
}
grid_search = GridSearchCV(DecisionTreeClassifier(), parameters, cv=5, verbose=True)
grid_search.fit(comments_train_X, comments_train_y)
print("Best params for Decision tree classifier are: ", grid_search.best_params_)
print("DTC with best params accuracy:", round(grid_search.score(comments_test_X, comments_test_y) * 100.0, 3), "%")
Fitting 5 folds for each of 300 candidates, totalling 1500 fits
Best params for Decision tree classifier are: {'ccp_alpha': 0.0, 'criterion': 'gini', 'max_depth': 19, 'min_samples_leaf': 1, 'random_state': 42}
DTC with best params accuracy: 93.112 %
data_tree_classifier = DecisionTreeClassifier(random_state=42, ccp_alpha=0.0, criterion='gini', max_depth=19, min_samples_leaf=1)
data_tree_classifier.fit(comments_train_X, comments_train_y)
print("DTC (random_state=42, ccp_alpha=0.0, criterion='gini', max_depth=19, min_samples_leaf=1) accuracy", round(data_tree_classifier.score(comments_test_X, comments_test_y) * 100.0, 3) , "%")
print("F1 score", f1_score(data_tree_classifier.predict(comments_test_X), comments_test_y, average=None))
confusion_matrix(data_tree_classifier)
DTC (random_state=42, ccp_alpha=0.0, criterion='gini', max_depth=19, min_samples_leaf=1) accuracy 93.112 %
F1 score [0.92520776 0.93617021]
from sklearn.model_selection import cross_val_score
cross_val_scores_dtc = cross_val_score(data_tree_classifier, comments_X, comments_y, cv=50)
print("50-fold cross validation average score for DTC: ", round(np.mean(cross_val_scores_dtc) * 100.0, 3), "%")
50-fold cross validation average score for DTC: 94.431 %
Logistic regression model
I přes jméno Logistická regrese, tak se tento model používá více ke klasifikaci nežli k regresi. Logistická regrese se také někdy uvádí v odborné literatuře jako Logit regression. Výsledek modelu jsou hodnoty 0 v případě, že jev nenastal a 1 v případě pokud ano, a které takto pak udávají výslednou pravděpodobnost.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
parameters = {
"random_state": [42],
"C": [0.2, 0.5, 1.0],
"class_weight": [None, "balanced"],
"solver": ["lbfgs", "liblinear"],
"max_iter": [50, 80, 100, 200, 500, 800, 1000],
}
grid_search = GridSearchCV(LogisticRegression(), parameters, cv=5, verbose=True)
grid_search.fit(comments_train_X, comments_train_y)
print("Best parameters are:", grid_search.best_params_)
print("Logistic regresion accuracy:", round(grid_search.score(comments_test_X, comments_test_y) * 100.0, 0), "%")
Fitting 5 folds for each of 84 candidates, totalling 420 fits
Best parameters are: {'C': 1.0, 'class_weight': None, 'max_iter': 50, 'random_state': 42, 'solver': 'liblinear'}
Logistic regresion accuracy: 96.0 %
logistic_regression = LogisticRegression(C = 1.0, class_weight = None, max_iter = 50, random_state = 42, solver = 'liblinear')
logistic_regression.fit(comments_train_X, comments_train_y)
print("Logistic regression model accuracy:", round(logistic_regression.score(comments_test_X, comments_test_y) * 100.0, 3) , "%")
print("F1 score", f1_score(logistic_regression.predict(comments_test_X), comments_test_y, average=None))
confusion_matrix(logistic_regression)
Logistic regression model accuracy: 95.918 %
F1 score [0.95604396 0.96190476]
from sklearn.model_selection import cross_val_score
cross_val_scores_lr = cross_val_score(logistic_regression, comments_X, comments_y, cv=50)
print("50-fold cross validation average score for Logistic regression model: ", round(np.mean(cross_val_scores_lr) * 100.0, 3), "%")
50-fold cross validation average score for Logistic regression model: 95.088 %
Support vector machines model
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
parameters = {'C': [0.01, 0.1, 1, 10.0, 100.0], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid']}
grid_search = GridSearchCV(SVC(), parameters, cv=5, verbose=True)
grid_search.fit(comments_train_X, comments_train_y)
print("Best params for SVM classifier model are: ", grid_search.best_params_)
print("SVM classifier with best params accuracy:", round(grid_search.score(comments_test_X, comments_test_y) * 100.0, 3), "%")
Fitting 5 folds for each of 20 candidates, totalling 100 fits
Best params for SVM classifier model are: {'C': 10.0, 'kernel': 'rbf'}
SVM classifier with best params accuracy: 95.408 %
support_vector_machine = SVC(C=10.0, kernel='rbf')
support_vector_machine.fit(comments_train_X, comments_train_y)
print("SVM model (rbf kernel, C=10) accuracy", round(support_vector_machine.score(comments_test_X, comments_test_y) * 100.0, 3) , "%")
print("F1 score", f1_score(support_vector_machine.predict(comments_test_X), comments_test_y, average=None))
confusion_matrix(support_vector_machine)
SVM model (rbf kernel, C=10) accuracy 95.408 %
F1 score [0.9494382 0.95794393]
from sklearn.model_selection import cross_val_score
from sklearn.metrics import f1_score
cross_val_scores_svm = cross_val_score(support_vector_machine, comments_X, comments_y, cv=50)
print("50-fold cross validation average score for SVM model: ", round(np.mean(cross_val_scores_svm) * 100.0, 3), "%")
50-fold cross validation average score for SVM model: 95.805 %
Kombinovaný model
from sklearn.ensemble import VotingClassifier
bayes = pipeline_multinomial_NB.predict(comments_test_X)
svc = support_vector_machine.predict(comments_test_X)
dtc = data_tree_classifier.predict(comments_test_X)
lr = logistic_regression.predict(comments_test_X)
estimators = [("bayes", pipeline_multinomial_NB), ("svm", support_vector_machine), ("decision trees", data_tree_classifier), ("logistic regression", logistic_regression)]
combined = VotingClassifier(estimators=estimators, weights=[1, 2, 1, 1])
combined.fit(comments_train_X, comments_train_y)
print("Combined model accuracy", round(combined.score(comments_test_X, comments_test_y) * 100.0, 3), "%")
print("F1 score", f1_score(combined.predict(comments_test_X), comments_test_y, average=None))
confusion_matrix(combined)
cross_val_scores_combined = cross_val_score(combined, comments_X, comments_y, cv=50)
print("50-fold cross validation score for combined model: ", round(np.mean(cross_val_scores_combined * 100.0), 3), "%")
Combined model accuracy 95.408 %
F1 score [0.95 0.95754717]
50-fold cross validation score for combined model: 96.267 %
Závěr
Porovnání modelů
cross_val_scores = pd.concat([
pd.DataFrame(cross_val_scores_nb * 100.0, columns=["Naive Bayes (Baseline)"]),
pd.DataFrame(cross_val_scores_dtc * 100.0, columns=["Decision Trees"]),
pd.DataFrame(cross_val_scores_lr * 100.0, columns=["Linear Regression"]),
pd.DataFrame(cross_val_scores_svm * 100.0, columns=["Support Vector Machines"]),
pd.DataFrame(cross_val_scores_combined * 100.0, columns=["Combined Model"])], axis=1)
fig, ax = plt.subplots(figsize=(12, 8))
sns.boxplot(data=pd.DataFrame(cross_val_scores), ax=ax)
Perličky aneb špatně určené zprávy
combined_prediction = combined.predict(comments_test_X)
predictions = pd.DataFrame(list(zip(comments_test_text, combined_prediction, comments_test_y)), columns =['Text', 'Combined Model', 'TRUE'])
missclasified_predictions = predictions.loc[(predictions['TRUE'] != predictions['Combined Model'])]
false_positives = missclasified_predictions.loc[(missclasified_predictions['Combined Model'] == 1) & (missclasified_predictions['TRUE'] == 0) ]
print("False positives (ham messages classified as spam):")
for index, row in false_positives.iterrows():
print(" ", row["Text"])
false_negatives = missclasified_predictions.loc[(missclasified_predictions['Combined Model'] == 0) & (missclasified_predictions['TRUE'] == 1) ]
print("\nFalse negatives (spam messages classified as ham):")
for index, row in false_negatives.iterrows():
print(" ", row["Text"])
False positives (ham messages classified as spam):
2,000,000,000 out of 7,000,000,000 people in the would saw this video just in 2 years and yeat i only get 2 words out of the hole song
How did THIS Video in all of YouTube get this many views and likes? Why Gangnam style? I don't have a problem with it, i just don't understand the phenomena behind it, it's just like any other random music video out there.
OMG I LOVE YOU KATY PARRY YOUR SONGS ROCK!!!!!!!!!!!!!!!!! THATS A TOTAL SUBSCRIBE
YOUTUBE MONEY !!!!!!!!!!!!!!!!!!!!!!!
Lemme Top Comments Please!!
False negatives (spam messages classified as ham):
Aslamu Lykum... From Pakistan
http://thepiratebay.se/torrent/6381501/Timothy_Sykes_Collection
los invito a subscribirse a mi canal
ｈｔｔｐ://ｗｗｗ.ｅｂａｙ.ｃｏｍ/ｕｓｒ/ｓｈｏｅｃｏｌｌｅｃｔｏｒ314
We can have a party next share
I love katy fashions tiger, care to visit my blog sinar jahitan I also have the tiger collections tqvm
Finally someone shares the same opinion as me. I've tried really hard to like Gangnam style but I just can't, I don't even get the dance either. A song like this should be more popular than Gangnam Style, at least it makes sense.
I'm A SUBSCRIBER
share your thoughts
She loves Vena. trojmiasto.pl/Vena-Bus-Taxi-o59253.html
http://thepiratebay.se/torrent/10626048/The.Expendables.3.2014.DVDScr.LEAKED.CLEAN.XviD.MP3-RARBG
http://thepiratebay.se/torrent/10626835/The.Expendables.3.2014.DVDSCR.Xvid-DiNGO
+447935454150 lovely girl talk to me xxx
data_spam_final_test = {'CONTENT': ["check out my new video, link below", "subscribe to my new channel", "check out my new website!"]}
data_ham_final_test = {'CONTENT': ["hi i really like your videos", "nice job i really enjoyed greatly", "can i ask when will be new video"]}
df_spam_final_test = pd.DataFrame(data=data_spam_final_test)
df_ham_final_test = pd.DataFrame(data=data_ham_final_test)
new_spam_comments = count_vectorizer.transform(df_spam_final_test["CONTENT"])
new_ham_comments = count_vectorizer.transform(df_ham_final_test["CONTENT"])
print("testovaní vytvořených spamových zpráv:", combined.predict(new_spam_comments))
print("testovaní vytvořených hamových zpráv:", combined.predict(new_ham_comments))
print("výsledek testování: 1 - jedná se o Spam, 0 - jedná se o Ham")
testovaní vytvořených spamových zpráv: [1 1 1]
testovaní vytvořených hamových zpráv: [0 0 0]
výsledek testování: 1 - jedná se o Spam, 0 - jedná se o Ham