TP Integrador de Big Data / Data Science
Gustavo Navarro - Comisión 22608 - Enero 2023
Análisis exploratorio
Carga de datos
# las primeras 5 filas
df.head()
idobject
genderobject
0
10-5894942
male
1
41-1676468
female
2
64-6396924
male
3
35-2426788
male
4
60-9387304
male
#Las últimas filas del dataframe
df.tail()
idobject
genderobject
1013
82-7312119
male
1014
45-3445439
male
1015
02-3651562
male
1016
05-5203587
female
1017
13-3347050
male
Resumen estadístico
df.describe()
math scorefloat64
physics scorefloat64
count
1011.0
1011.0
mean
66.48071216617211
69.06330365974283
std
15.326879704379337
14.694107007851635
min
13.0
27.0
25%
56.0
60.0
50%
67.0
70.0
75%
77.0
79.0
max
100.0
100.0
#Revisar los tipos de datos
df.dtypes
Limpieza y ordenamiento de datos
Encontrar las filas duplicadas
print(f'Original: {df.id.count()} filas')
duplicate_rows_df = df[df.duplicated()]
print(f'Cantidad de filas duplicadas: {duplicate_rows_df.id.count()}')
#Eliminar los duplicados
df = df.drop_duplicates()
Original: 1018 filas
Cantidad de filas duplicadas: 18
#Cantidad de filas despues de eliminar los duplicados
print(f'Final: {df.id.count()} filas')
Final: 1000 filas
#Eliminar las columnas irrelevantes
print(df.columns)
df =df.drop(['id'], axis=1)
Index(['id', 'gender', 'race/ethnicity', 'parental level of education',
'lunch', 'employed', 'test preparation course', 'math score',
'physics score', 'chemistry score', 'algebra_score'],
dtype='object')
print(df.columns)
Index(['gender', 'race/ethnicity', 'parental level of education', 'lunch',
'employed', 'test preparation course', 'math score', 'physics score',
'chemistry score', 'algebra_score'],
dtype='object')
Renombrar las columnas
df = df.rename(columns = {
'gender': 'Gender',
'race/ethnicity': 'Ethnicity',
'parental level of education': 'Parental level of education',
'lunch': 'Lunch',
'employed': 'Employed',
'test preparation course': 'Test preparation course',
'math score': 'Math score',
'physics score': 'Physics score',
'chemistry score': 'Chemistry score',
'algebra_score': 'Algebra score'})
df.columns
Eliminación de valores perdidos o nulos
#Encontrar los valores nulos
print(df.isnull().sum())
#Eliminar los valores perdidos
df = df.dropna()
print()
#Despues de eliminar los nulos
print(df.isnull().sum())
Gender 0
Ethnicity 0
Parental level of education 0
Lunch 0
Employed 0
Test preparation course 0
Math score 7
Physics score 7
Chemistry score 7
Algebra score 7
dtype: int64
Gender 0
Ethnicity 0
Parental level of education 0
Lunch 0
Employed 0
Test preparation course 0
Math score 0
Physics score 0
Chemistry score 0
Algebra score 0
dtype: int64
Detectar los valores outliers
sns.boxplot(x=df['Math score'])
plt.show()
sns.boxplot(x=df['Physics score'])
plt.show()
sns.boxplot(x=df['Algebra score'])
plt.show()
sns.boxplot(x=df['Chemistry score'])
plt.show()
print(f'Antes: {df.Lunch.count()} filas\n')
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
df = df[~((df < (Q1 - 1.5*IQR)) | (df > (Q3 + 1.5*IQR))).any(axis=1)]
print(f'\nDespues: {df.Lunch.count()} filas')
Antes: 993 filas
Math score 21.0
Physics score 19.0
Chemistry score 21.0
Algebra score 19.0
dtype: float64
Despues: 984 filas
/tmp/ipykernel_156/434115097.py:6: FutureWarning: Automatic reindexing on DataFrame vs Series comparisons is deprecated and will raise ValueError in a future version. Do `left, right = left.align(right, axis=1, copy=False)` before e.g. `left == right`
df = df[~((df < (Q1 - 1.5*IQR)) | (df > (Q3 + 1.5*IQR))).any(axis=1)]
/tmp/ipykernel_156/434115097.py:6: FutureWarning: Automatic reindexing on DataFrame vs Series comparisons is deprecated and will raise ValueError in a future version. Do `left, right = left.align(right, axis=1, copy=False)` before e.g. `left == right`
df = df[~((df < (Q1 - 1.5*IQR)) | (df > (Q3 + 1.5*IQR))).any(axis=1)]
Visualización de los datos obtenidos
Encontrar correlaciones y frecuencias
plt.hist(df['Algebra score'], bins=20)
plt.title('Algebra')
plt.ylabel('Student count')
plt.xlabel('Score')
plt.show()
plt.hist(df['Chemistry score'], bins=20)
plt.title('Chemistry')
plt.ylabel('Student count')
plt.xlabel('Score')
plt.show()
plt.hist(df['Math score'], bins=20)
plt.title('Math')
plt.ylabel('Student count')
plt.xlabel('Score')
plt.show()
plt.hist(df['Physics score'], bins=20)
plt.title('Physics')
plt.ylabel('Student count')
plt.xlabel('Score')
plt.show()
#Correlacion entre los datos = mapa de calor
c = df.corr()
print(c)
Math score Physics score Chemistry score Algebra score
Math score 1.000000 0.812055 0.798312 0.916674
Physics score 0.812055 1.000000 0.951536 0.968358
Chemistry score 0.798312 0.951536 1.000000 0.964652
Algebra score 0.916674 0.968358 0.964652 1.000000
plt.figure(figsize= (20, 10))
sns.heatmap(c, cmap='BrBG', annot=True)
plt.show()
#pandas.value_counts()-> devuelve una serie de valores unicos en un orden descendente de frecuencia
labels = df['Gender'].value_counts().index
sizes = df['Gender'].value_counts()
plt.pie(sizes, labels=labels, autopct='%1.2f%%')
plt.title('Gender')
plt.show()
#Repetimos para Etnia, empleo y el resto de las variables categóricas
labels = df['Ethnicity'].value_counts().index
sizes = df['Ethnicity'].value_counts()
plt.pie(sizes, labels=labels, autopct='%1.2f%%')
plt.title('Ethnicity')
plt.show()
labels = df['Employed'].value_counts().index
sizes = df['Employed'].value_counts()
plt.pie(sizes, labels=labels, autopct='%1.2f%%')
plt.title('Employed')
plt.show()
labels = df['Test preparation course'].value_counts().index
sizes = df['Test preparation course'].value_counts()
plt.pie(sizes, labels=labels, autopct='%1.2f%%')
plt.title('Test preparation course')
plt.show()
labels = df['Parental level of education'].value_counts().index
sizes = df['Parental level of education'].value_counts()
plt.pie(sizes, labels=labels, autopct='%1.2f%%')
plt.title('Parental level of education')
plt.show()
labels = df['Lunch'].value_counts().index
sizes = df['Lunch'].value_counts()
plt.pie(sizes, labels=labels, autopct='%1.2f%%')
plt.title('Lunch')
plt.show()
Respondiendo preguntas
¿Hay alguna relación entre el promedio de notas obtenidas y el hecho de haber realizado el curso preparatorio?
#axis=1 hace que se aplique la función sobre los valores numericos de la fila en vez de las columnas
df['Average score'] = df.mean(axis=1)
df
Genderobject
male51.8%
female48.2%
Ethnicityobject
group C32.2%
group D26.3%
3 others41.5%
0
male
group A
1
female
group D
2
male
group E
3
male
group B
4
male
group E
5
female
group D
6
female
group A
7
male
group E
8
male
group D
9
male
group C
si = df[df['Test preparation course'] == 'completed']
no = df[df['Test preparation course'] == 'none']
Cantidad de alumnos que hicieron y que no hicieron el curso preparatorio
print('Realizaron el curso:', si['Test preparation course'].count())
print('No realizaron el curso:', no['Test preparation course'].count())
Realizaron el curso: 332
No realizaron el curso: 652
sns.histplot(si['Average score'], color = 'green', alpha= .4, fill = True)
sns.histplot(no['Average score'], color = 'red', alpha= .4, fill = True)
plt.show()
Promedio de notas de los alumnos que hicieron y no hicieron el curso preparatorio
df.drop(['id', 'gender','race/ethnicity', 'parental level of education', 'lunch', 'employed'], axis = 'columns', inplace=True)
df
Execution Error
KeyError: "['id' 'gender' 'race/ethnicity' 'parental level of education' 'lunch'\n 'employed'] not found in axis"
df =df.round(2)
df
Genderobject
male51.8%
female48.2%
Ethnicityobject
group C32.2%
group D26.3%
3 others41.5%
0
male
group A
1
female
group D
2
male
group E
3
male
group B
4
male
group E
5
female
group D
6
female
group A
7
male
group E
8
male
group D
9
male
group C
df.groupby(['test preparation course']).mean().round(2)
math scorefloat64
physics scorefloat64
completed
69.84
74.18
none
64.78
66.47
Conclusión
Si bien la cantidad de alumnos que no realizaron el curso duplicaron a los que si lo hicieron, esta diferencia no se ve reflejada significativamente en el promedio de las notas. Dicha diferencia es entre 5 y 10 puntos según la materia. Lo cual nos indica que no es necesariamente obligatorio el curso preparatorio.
Se recomienda auditar el contenido del curso, a fin de lograr un mejor rendimiento en el nivel académico e incrementar el interés del alumnado.