Analysis

Loading data

import pandas as pd import seaborn as sns import numpy as np import matplotlib.pyplot as plt import plotly.express as px

df = pd.read_csv('DataScientist.csv')

Unnamed: 0int64

0 - 3908

indexint64

0 - 4379

df.columns

df['Job Title'].value_counts()

Cleaning Data

jobs_list = ['Data Scientist', 'Data Engineer','Data Analyst'] df_p = df[df['Job Title'].isin(jobs_list)] df_p

Unnamed: 0int64

3 - 3853

indexint64

3 - 4324

Is it possible predict salary based on location? [In progress]

df_pred = df_p[['Job Title', 'Salary Estimate','Location']] df_pred

Job Titleobject

Data Scientist35.1%

Data Engineer33.3%

Data Analyst31.5%

Salary Estimateobject

$148K-$174K (Glassdoor est.)2.8%

$113K-$180K (Glassdoor est.)2.7%

116 others94.5%

Data Analyst

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$120K-$140K (Glassdoor est.)

Data Scientist

$120K-$140K (Glassdoor est.)

df_pred['Salary Estimate'].value_counts()

df_pred['PerHour'] = df_pred["Salary Estimate"].str.find("Per Hour") df_pred.sort_values(by='PerHour', ascending=False)

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

Job Titleobject

Data Scientist35.1%

Data Engineer33.3%

Data Analyst31.5%

Salary Estimateobject

$148K-$174K (Glassdoor est.)2.8%

$113K-$180K (Glassdoor est.)2.7%

116 others94.5%

689

Data Engineer

$34-$53 Per Hour(Glassdoor est.)

Data Analyst

$111K-$181K (Glassdoor est.)

2480

Data Scientist

$12K-$112K (Glassdoor est.)

2274

Data Engineer

$71K-$122K (Glassdoor est.)

2280

Data Engineer

$71K-$122K (Glassdoor est.)

2285

Data Engineer

$71K-$122K (Glassdoor est.)

2291

Data Engineer

$39K-$69K (Glassdoor est.)

2466

Data Scientist

$12K-$112K (Glassdoor est.)

2468

Data Scientist

$12K-$112K (Glassdoor est.)

2473

Data Scientist

$12K-$112K (Glassdoor est.)

df_pred = df_pred[df_pred.PerHour != 8] df_pred.sort_values(by='PerHour', ascending=False)

Job Titleobject

Data Scientist35.2%

Data Engineer33.2%

Data Analyst31.6%

Salary Estimateobject

$148K-$174K (Glassdoor est.)2.8%

$113K-$180K (Glassdoor est.)2.7%

115 others94.5%

Data Analyst

$111K-$181K (Glassdoor est.)

2512

Data Scientist

$12K-$111K (Glassdoor est.)

2267

Data Engineer

$71K-$122K (Glassdoor est.)

2274

Data Engineer

$71K-$122K (Glassdoor est.)

2280

Data Engineer

$71K-$122K (Glassdoor est.)

2285

Data Engineer

$71K-$122K (Glassdoor est.)

2291

Data Engineer

$39K-$69K (Glassdoor est.)

2466

Data Scientist

$12K-$112K (Glassdoor est.)

2468

Data Scientist

$12K-$112K (Glassdoor est.)

2473

Data Scientist

$12K-$112K (Glassdoor est.)

Transforming salary estimate from string to int

import re as re def find_number(text): num = re.findall(r'[0-9]+',text) return " ".join(num) df_pred['Salaries']=df_pred['Salary Estimate'].apply(lambda x: find_number(x)) df_pred.head()

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """

Job Titleobject

Salary Estimateobject

Data Analyst

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

df_pred[['InicialSalary','FinalSalary']] = df_pred['Salaries'].str.split(' ', expand=True).astype(str).astype(int) df_pred.head()

/shared-libs/python3.7/py/lib/python3.7/site-packages/pandas/core/frame.py:3191: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[k1] = value[k2]

Job Titleobject

Salary Estimateobject

Data Analyst

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

df_pred['Mean'] = df_pred[['InicialSalary', 'FinalSalary']].mean(axis=1) df_pred

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

Job Titleobject

Data Scientist35.2%

Data Engineer33.2%

Data Analyst31.6%

Salary Estimateobject

$148K-$174K (Glassdoor est.)2.8%

$113K-$180K (Glassdoor est.)2.7%

115 others94.5%

Data Analyst

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$111K-$181K (Glassdoor est.)

Data Scientist

$120K-$140K (Glassdoor est.)

Data Scientist

$120K-$140K (Glassdoor est.)

df_pred = df_pred.drop(['Salary Estimate','PerHour','Salaries'], axis = 1)

df_pred[['City','State']] = df_pred['Location'].str.split(', ', expand=True) df_pred.head()

Job Titleobject

Locationobject

Data Analyst

Lyndhurst, NJ

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

df_pred['NormStates'] = pd.factorize(df_pred['State'])[0] df_pred

Job Titleobject

Data Scientist35.2%

Data Engineer33.2%

Data Analyst31.6%

Locationobject

New York, NY11.3%

Chicago, IL11%

92 others77.7%

Data Analyst

Lyndhurst, NJ

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

Data Scientist

New York, NY

All positions

from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression x = df_pred.iloc[:,7].values y = df_pred.iloc[:,4].values

X_train, X_test, Y_train, Y_test = train_test_split(x,y, test_size=0.2, random_state = 0) X_train

X_train= X_train.reshape(-1, 1) Y_train= Y_train.reshape(-1, 1) Y_test = X_test.reshape(-1, 1) X_test = X_test.reshape(-1, 1)

regressor = LinearRegression() regressor.fit(X_train, Y_train)

viz_train = plt viz_train.scatter(X_train, Y_train, color = 'blue') viz_train.plot(X_train, regressor.predict(X_train), color = 'black') viz_train.title('Salario vs Ciudad') viz_train.xlabel('Ciudad') viz_train.ylabel('Salario') viz_train.show()

regressor.score(X_test, Y_test)

Job offers by state

states = pd.DataFrame(df_pred['State'].value_counts().reset_index().values, columns=['State','Total']) states['Total'] = states['Total'].astype(str).astype(int) #states['Two Letter State'] = states['Two Letter State'].apply(lambda x: x.strip()) states

Stateobject

TX10%

CA10%

8 others80%

Totalint64

2 - 220

220

213

100

from states import code d = {key: value for (value, key) in code.items()} states['State Name'] = states['State'].map(d)

fig = px.choropleth(states, locations='State', color='Total', color_continuous_scale='spectral_r', hover_name='State Name', locationmode='USA-states', labels={'Total':'Total'}, scope='usa') fig.add_scattergeo( locations=states['State'], locationmode='USA-states', text=states['State'], mode='text', hoverinfo='skip') fig.update_layout( title={'text':'Total Job Offers by State', 'xanchor':'center', 'yanchor':'top', 'x':0.5}) fig.show()

Finding relevant words from job descriptions

df_test = df_p

import nltk nltk.download('punkt') nltk.download('averaged_perceptron_tagger') from nltk.tokenize import sent_tokenize, word_tokenize

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!

def find_verb(keyword): tokens = nltk.word_tokenize(keyword) tagged = nltk.pos_tag(tokens) verb = [w for w,t in tagged if "VB" in t] if len(verb) < 1: return "" else: return verb def find_adjective(keyword): tokens = nltk.word_tokenize(keyword) tagged = nltk.pos_tag(tokens) adjective = [w for w,t in tagged if "JJ" in t] if len(adjective) < 1: return "" else: return adjective def find_noun(keyword): tokens = nltk.word_tokenize(keyword) tagged = nltk.pos_tag(tokens) noun = [w for w,t in tagged if "NN" in t] if len(noun) < 1: return "" else: return noun df_test['Verb'] = df_test['Job Description'].apply(find_verb) df_test['Adjective'] = df_test['Job Description'].apply(find_adjective) df_test['Noun'] = df_test['Job Description'].apply(find_noun)

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:28: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:29: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:30: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

df_test['Verb'] = [' '.join(map(str, l)) for l in df_test['Verb']] df_test['Adjective'] = [' '.join(map(str, l)) for l in df_test['Adjective']] df_test['Noun'] = [' '.join(map(str, l)) for l in df_test['Noun']] df_test.head()

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

Unnamed: 0int64

indexint64

from collections import Counter Counter(" ".join(df_test["Adjective"]).split()).most_common(50)

Counter(" ".join(df_test["Noun"]).split()).most_common(50)

Counter(" ".join(df_test["Verb"]).split()).most_common(50)

Adding a status

options = ['Applied','Applied','Applied','Applied', 'Technical Interview', 'Technical Interview', 'Technical Interview', 'First Interview', 'First Interview', 'First Interview','Second Interview','Second Interview','Rejected','Job Offered'] df_status = df_p df_status['Status'] = np.random.choice(list(options), len(df_status)) df_status.head(5)

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning:


A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

Unnamed: 0int64

indexint64

freq = df_status['Status'].value_counts() freq

sns.set(rc={'figure.figsize':(13,9)}) sns.histplot(df_status['Status'])