import numpy as np import pandas as pd import matplotlib.pyplot as plt from pandas.plotting import parallel_coordinates %matplotlib inline

missing_values = [" ?"] data = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data',header=None, na_values = missing_values) data.columns = ['Age', 'Work-Class', 'Fnlwgt', 'Education', 'Education-Num', 'Marital-Status', 'Occupation', 'Relationship', 'Race', 'Sex', 'Capital-Gain', 'Capital-Loss', 'Hours-Per-Week', 'Native-Country', 'Num'] data

Part1

1) For each continuous attribute, calculate its average, standard deviation, minimum, and maximum values.

continousCol = ['Age', "Fnlwgt", 'Education-Num', 'Capital-Gain', 'Capital-Loss', 'Hours-Per-Week'] applyFunc = lambda f: [(x, f(data[x])) for x in continousCol] print('Average') print(applyFunc(np.average)) print('Standard Deviation') print(applyFunc(np.std)) print('Min Value') print(applyFunc(np.min)) print('Max Value') print(applyFunc(np.max))

2) For the discrete attribute, count the frequency for each of its distinct values.

def counts(index): counts = {} for x in data[index]: counts[x] = counts.get(x, 0) + 1 return counts discreteCol = ["Work-Class", "Education", "Marital-Status", "Occupation", "Relationship", "Race", "Sex", "Native-Country"] commons = dict() for x in discreteCol: c = counts(x) print("{}: {}".format(x, c)) commons[x] = max(c, key = lambda y: c[y])

3) Draw histogram of the class variable

fig, axes = plt.subplots(nrows=2, ncols=len(discreteCol)//2, figsize=(25,10)) for n, x in enumerate(discreteCol): pd.Series(data[x]).value_counts().plot(kind='bar', ax=axes[n//4, n%4])

4) Draw the distribution of values for a continuous attribute using a histogram.

fig, axes = plt.subplots(nrows=2, ncols=len(continousCol)//2, figsize=(25,10)) for n, x in enumerate(continousCol): axes[n//3, n%3].hist(data[x])

5) Draw some scatter plots for a couple of attribute pairs.

fig,a = plt.subplots(1,2, figsize=(25,10)) a[0].scatter(data['Age'], data['Capital-Gain']) a[1].scatter(data['Age'], data['Capital-Loss']) plt.show()

6) Draw a parallel diagram for some attributes in the data set

def plotParallelCoordinates(col): data2 = data.drop(filter(lambda x: x != col, discreteCol + ["Num", "Fnlwgt"]), axis=1) return parallel_coordinates(data2, col)

plotParallelCoordinates("Marital-Status")

plotParallelCoordinates("Education")

plotParallelCoordinates("Relationship")

7) For each diagram describe your interpretation and insight.

Insights:

Our first insight was looking at the age distribution, we noticed that the dataset was heavily skewed towards the younger population, with the vast majority of entries being 40 and under. This skew towards younger demographics could potnetially skew other data values, so we will keep this in mind. The next observation was that the captial gains / capital loss was 0 for a vast majority of people. We believe this may be due to the younger age demographic skew we discussed earlier, as younger people are less likely on average to be invested in the stock market, which would cause 0 captical gains/losses.

We noticed a large spike in "hours worked per week" at the 40 mark. This was a good sanity check to confirm that our advanced data analysis techniques were working as intended, since the majority of working people are working 40 hours a week. Another interesting skew we discovered was that almost every participant was located in the United States. This will also potentially skew the results we discover.

For out scatter plot we decided to analyze the age correlation with captial gains/losses to confirm our hypothesis. As it turns out we were actually incorrect in our original theory. This was a great discovery, because it showed that our preconcieved ideas were not accurate. As it turns out, there is not a strong correlation between age and captial gains/loss, which seems to show that people of all ages dabble with investments. This discovery showed the power of data analysis

Part 2

Identify which attributes have missing values and address the issue by:

1) Replacing missing values by the average or mod of the attribute (based on attribute types)

data2 = data.replace(' ?', np.NaN).drop("Num", axis=1) for x in discreteCol: data2[x][data2[x].isna()] = commons[x] print(data2.isna().any())

2) Replace missing values by the average or mode of the attribute in the particular class to which the instance belongs

fig, axes = plt.subplots(nrows=2, ncols=len(discreteCol)//2, figsize=(25,10)) for n, x in enumerate(discreteCol): pd.Series(data[x]).value_counts().plot(kind='bar', ax=axes[n//4, n%4])

3) Draw a histogram of the attribute before and after replacing missing values in the previous step 1 and 2

fig, axes = plt.subplots(nrows=2, ncols=len(discreteCol)//2, figsize=(25,10)) for n, x in enumerate(discreteCol): pd.Series(data2[x]).value_counts().plot(kind='bar', ax=axes[n//4, n%4])