Homework 2: Data Cleaning Basics

In this homework, we will be going over a couple of concepts we learned in lecture 2! Everything in this homework was covered in lecture so feel free to reference the slides and remember you can also google!

Lecture slides: https://docs.google.com/presentation/d/1TIML5REJKThU4XVHe28m2-zy-NGByi-9Y8upwGGPGrs/edit#slide=id.g215ffca71a4_0_1

import pandas as pd import numpy as np from datascience import *

movies = pd.read_csv("movies_dataset.csv") movies.head(10)

Question 1:

#In the table above, there are nan values. Normally we would investigate more before #removing these rows but for the sake of this homework let us just remove them! Write a line of code below #in "nans_removed" to remove all nan values. nans_removed = movies.dropna() nans_removed

Question 2:

#Now that we've removed the nan values let's remove some columns. Suppose the columns: {"writer", "storyline", # "industry"} don't interest us. Write a line of code in "columns_dropped" that removes these columns columns_dropped = nans_removed.drop(columns=['writer', 'storyline', 'industry']) columns_dropped

Question 3:

#We've decided to focus on more popular movies. So we want to only keep movies that have an #IMDb-rating of at least 5. Write a line of code in "popular_only" to do this! #Reference the lecture slides if you're stuck! popular_only = columns_dropped[columns_dropped['IMDb-rating'] >= 5] popular_only

Question 4

#For this question please respond with a text cell outlining the steps involved in Data Cleaning. #Please also include steps for what you would do with outliers, missing/NA values and wrong type values.

- For outliners, investigate then might remove them. - For missing / NA values or wrong values, I could try valrious methods to replace the the missing / NA by mean, median, or linear regression.

CONGRATS! You've finished the coding part of your homework. For the last part of your homework include a summary of 5 things you learned from this week's DSS's Article:

Link: https://medium.com/@dssberkeley/chatgpt-a-teachers-worst-nightmare-or-the-future-of-education-be6d6014a64a

ChatGPT could be a great study buddy or essay reviewer.

Banning AI is not a solution because people always try to game the system.

Using AI while in school will prepare students for the real world.

Teachers can use ChatGPT to improve their materials.

There are other AIs that do the same as ChatGPT, and will be more of them. Banning is not the solution.