2nd Assignment - UML with Pokemon
# Standard packaging
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
# Load the dataset
pokemon = pd.read_csv("https://sds-aau.github.io/SDS-master/00_data/pokemon.csv")
1. Give a brief overview of data, what variables are there, how are the variables scaled and variation of the data columns
To give an overview of the data, we use the general functions: head, info and describe. We can see that the dataset contains integers, objects/strings and boolean type of data. The variables important for the dimensionality reduction are between the columns Total - Generation. We can also see that the variables have different scales and we don't know the metrics for each column. The mean and standard deviation for each variable is very different.
# Check the data
pokemon.head()
# Check the data
pokemon.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 800 entries, 0 to 799
Data columns (total 13 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Number 800 non-null int64
1 Name 800 non-null object
2 Type1 800 non-null object
3 Type2 414 non-null object
4 Total 800 non-null int64
5 HitPoints 800 non-null int64
6 Attack 800 non-null int64
7 Defense 800 non-null int64
8 SpecialAttack 800 non-null int64
9 SpecialDefense 800 non-null int64
10 Speed 800 non-null int64
11 Generation 800 non-null int64
12 Legendary 800 non-null bool
dtypes: bool(1), int64(9), object(3)
memory usage: 75.9+ KB
#Check the data
pokemon.describe()
2. Execute a PCA analysis on all numerical variables in the dataset. Hint: Don’t forget to scale them first. Use 4 components. What is the cumulative explained variance ratio? Hint: I am not sure this terminology and code was introduced during class, but try and look into cumulative explained variance and sklearn(package) and see if you can figure out the code needed.
# Scale the data
# Load up Standard Scaler from sklearn
from sklearn.preprocessing import StandardScaler
# And scale all relevant variables into a new matrix
scaled_pokemon = StandardScaler().fit_transform(pokemon.loc[:,'Total':'Generation'])
# All variables now have a mean of 0 and std of 1
for i in range(8):
sns.distplot(scaled_pokemon[:,i], hist=False)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `kdeplot` (an axes-level function for kernel density plots).
warnings.warn(msg, FutureWarning)
# Execute PCA analysis on all numerical variables
# Import PCA
from sklearn.decomposition import PCA
pca = PCA(n_components=4)
# Use PCA to transform the data
pca_pokemon = pca.fit_transform(scaled_pokemon)
pca_pokemon.shape
sns.scatterplot(pca_pokemon[:,0], pca_pokemon[:,1], hue = pokemon['Type1'] )
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
FutureWarning
We can compare the variance in the overall dataset to what was captured from the two primary components using .explained_variance_ratio_. We can see that the first four principal components explain the majority of the dataset: 82,31%. This indicates the total information represented compared to the original data.
#Variance of each component
print('Variance of each component:', pca.explained_variance_ratio_)
#Total variance explained
print('Total variance explained:', sum(list(pca.explained_variance_ratio_))*100)
Variance of each component: [0.46410425 0.13876986 0.12336467 0.09686493]
Total variance explained: 82.3103702693512
3. Use a different dimensionality reduction method (eg. UMAP/NMF) – do the findings differ?
UMAP
We can see on the scatter plots that it looks different from the above one.
!pip install -q umap-learn
WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
You should consider upgrading via the '/root/venv/bin/python -m pip install --upgrade pip' command.
import umap
reducer = umap.UMAP()
2021-09-20 08:21:24.302779: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-09-20 08:21:24.302823: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
umap_pokemon = reducer.fit_transform(scaled_pokemon)
umap_pokemon.shape
sns.scatterplot(umap_pokemon[:,0], umap_pokemon[:,1], hue = pokemon['Type1'] )
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
FutureWarning
4. Perform a cluster analysis (KMeans) on all numerical variables (scaled & before PCA). Pick a realistic number of clusters (up to you where the large clusters remain mostly stable).
from sklearn.cluster import KMeans
# Check the ideal number of clusters, use the 'elbow rule' for deciding
ks = range(1, 6)
inertias = []
for k in ks:
# Create a KMeans instance with k clusters: model
model = KMeans(n_clusters=k)
# Fit model to samples
model.fit(scaled_pokemon)
# Append the inertia to the list of inertias
inertias.append(model.inertia_)
# Plot ks vs inertias
plt.plot(ks, inertias, '-o')
plt.xlabel('number of clusters, k')
plt.ylabel('inertia')
plt.xticks(ks)
plt.show() # --> 3 looks like a good number of clusters
clusterer = KMeans(n_clusters=3)
clusterer.fit(scaled_pokemon)
sns.scatterplot(scaled_pokemon[:,0], scaled_pokemon[:,1], hue = clusterer.labels_)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
FutureWarning
5. Visualise the first 2 principal components and color the datapoints by cluster.
sns.set(color_codes=True, rc={'figure.figsize':(10,8)})
sns.scatterplot(pca_pokemon[:,0], pca_pokemon[:,1], hue = clusterer.labels_)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
FutureWarning
6. Inspect the distribution of the variable Type1 across clusters. Does the algorithm separate the different types of pokemon?
The type of the Pokemon does not define the stats of itself, thus it makes sense that the distribution of the type is not good across clusters and the types of Pokemon is not separated so well.
# we can check out a cross-tab
pd.crosstab(clusterer.labels_, pokemon['Type1'])
7. Perform a cluster analysis on all numerical variables scaled and AFTER dimensionality reduction and visualize the first 2 principal components.
clusterer = KMeans(n_clusters=2)
clusterer.fit(umap_pokemon)
sns.scatterplot(umap_pokemon[:,0], umap_pokemon[:,1], hue = clusterer.labels_)
/shared-libs/python3.7/py/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
FutureWarning
8. Again, inspect the distribution of the variable “Type 1” across clusters, does it differ from the distribution before dimensionality reduction?
It is somewhat different than above, but as mentioned before, the variables do not define or rank the type of Pokemon so well, thus it is not separated well across the clusters.
pd.crosstab(clusterer.labels_, pokemon['Type1'])