Data Science with Python
This notebook contains the data science applications possible in python and starts with basic introduction to python language from data science utilization perspective to dealing with data manipulation. Its divided intro 4 parts shown below
- Python Basics (Data types, storing values taking user input, string manipulation, mathematical operations) - Python Data Structures ( String, Arrays, Lists, Tuple, Dictionary) - Python Condition Programming ( OOPS, conditional statements, exception handling) - Python Data Manipulation (Read & Write files, data manipulations, statistical analysis, sampling, etc.)
Python Basics
Let's start by taking input from the user
user_given = input("Please enter your username : ")
print("Username given by user is", user_given)
Username given by user is Aditya
Now taking multiple inputs from the user
x, y = input("Enter two values: ").split()
print("Two inputs provided by the user are", x, y)
Two inputs provided by the user are 12 45
Variables in python along with some basic operations on them
a = 10
print("Value of integer variable a is",a)
# here a is the variable of integer type storing 10 as value
Value of integer variable a is 10
b = 30
# addition operation
print("Addition result for variables a and b is",a + b)
Addition result for variables a and b is 40
# substraction operation
print("Substraction result for variables a and b is",a - b)
Substraction result for variables a and b is -20
# multiplication operation
print("Multiplication result for variables a and b is",a * b)
Multiplication result for variables a and b is 300
# divison operation
print("Divison result for variables a and b is",a / b)
Divison result for variables a and b is 0.3333333333333333
# modulus operation
print("Modulus result for variables a and b is",a % b)
Modulus result for variables a and b is 10
Operators
# Examples of Arithmetic Operator
# Addition of numbers
add = a + b
# Subtraction of numbers
sub = a - b
# Multiplication of number
mul = a * b
# Division(float) of number
div1 = a / b
# Division(floor) of number
div2 = a // b
# Modulo of both number
mod = a % b
# Power
p = a ** b
# print results
print(add)
print(sub)
print(mul)
print(div1)
print(div2)
print(mod)
print(p)
40
-20
300
0.3333333333333333
0
10
1000000000000000000000000000000
# Examples of Relational Operators
# a > b is False
print(a > b)
# a < b is True
print(a < b)
# a == b is False
print(a == b)
# a != b is True
print(a != b)
# a >= b is False
print(a >= b)
# a <= b is True
print(a <= b)
False
True
False
True
False
True
# Examples of Logical Operator
a1 = True
b1 = False
# Print a and b is False
print(a1 and b1)
# Print a or b is True
print(a1 or b1)
# Print not a is False
print(not a1)
False
True
False
# Examples of Bitwise operators
# Print bitwise AND operation
print(a & b)
# Print bitwise OR operation
print(a | b)
# Print bitwise NOT operation
print(~a)
# print bitwise XOR operation
print(a ^ b)
# print bitwise right shift operation
print(a >> 2)
# print bitwise left shift operation
print(a << 2)
10
30
-11
20
2
40
# Examples of Assignment Operators
# Assign value
b = a
print(b)
# Add and assign value
b += a
print(b)
# Subtract and assign value
b -= a
print(b)
# multiply and assign
b *= a
print(b)
# bitwise lishift operator
b <<= a
print(b)
10
20
10
100
102400
#if elif and else for keywords condition checking , if checks if the condition stated in it is true and elif works when if is false and check the condition in elif and both condition don't meet then else will occur
a = 5
b = 15
if b % a == 0 :
print ("b is divisible by a")
elif b + 1 == 10:
print ("Increment in b produces 10")
else:
print ("You are in else statement")
b is divisible by a
Python Data Structures
Discussing about various data structures present in python and how to use and manipulate them#
#Strings
"medium_is_nice" # can be declared like this
'medium_is_nice' # or this
''' medium is nice ''' # or this
'''
medium
is nice
'''
# or like this also
# Strings are unltimately collection of characters
string1 = "Medium is nice"
print("Printing first element of string1 is",string1[0])
print("Printing last element of string1 is",string1[-1])
Printing first element of string1 is M
Printing last element of string1 is e
# reverse the string
print(string1[::-1])
ecin si muideM
# slicing and printing the string from 4rd and 10th character
print(string1[4:10])
um is
# slicing and printing the string 3rd and 2nd last character
print(string1[4:-2])
um is ni
# deleting the string and starting the string from index 3
print(string1[3:])
# deleting whole string
del string1
ium is nice
List data structure in python are like dynamically typed arrays.
# List
variable1 = ['Medium','is','fun'] # one type of list
varibale2 = [1,'Medium',2,'fun'] # another type of list
print("List 1 is",variable1,"List 2 is",varibale2)
List 1 is ['Medium', 'is', 'fun'] List 2 is [1, 'Medium', 2, 'fun']
# accessing list elements
print(variable1[0]) #accessing first element of list
print(varibale2[2]) #accessing third element of list
Medium
2
# getting size of list
print(len(variable1))
3
# adding element to list
variable1.append(".")
print(variable1)
['Medium', 'is', 'fun', '.']
# adding elements to list
variable1.extend(["best","articles","are","here"])
print(variable1)
['Medium', 'is', 'fun', '.', 'best', 'articles', 'are', 'here']
# reversing a list
variable1.reverse()
print(variable1)
['here', 'are', 'articles', 'best', '.', 'fun', 'is', 'Medium']
# removing elements from a list
variable1.remove('best')
print(variable1)
['here', 'are', 'articles', '.', 'fun', 'is', 'Medium']
Tuples in python are collections just like above list data structure, any data type can be stored in tuple and are indexed by integers
tuple1 = ("Medium","the","best")
print(tuple1)
('Medium', 'the', 'best')
# reversing tuple
print(tuple1[::-1])
('best', 'the', 'Medium')
# deleting tuple
del tuple1
Set in python is unordered collection data structure of data type that are iterative.
# creating a set
set1 = set("medium")
set2 = {'m','a','i'}
print("Set 1 elements are", set1," and Set 2 elements are", set2)
Set 1 elements are {'e', 'i', 'm', 'u', 'd'} and Set 2 elements are {'m', 'a', 'i'}
# adding element to set
set1.add("best")
print(set1)
{'e', 'i', 'best', 'm', 'u', 'd'}
# adding elements to set
set1.update("site","for","blog")
print(set1)
{'r', 't', 'o', 'e', 'l', 's', 'i', 'g', 'f', 'best', 'm', 'u', 'b', 'd'}
# accessing set elements since its unordered need to use loop
for i in set1:
print(i)
r
t
o
e
l
s
i
g
f
best
m
u
b
d
# removing elements from set
set1.remove('best')
print(set1)
{'r', 't', 'o', 'e', 'l', 's', 'i', 'g', 'f', 'm', 'u', 'b', 'd'}
Dictionary in Python is the data structure that is collection of keys values used for data values like map and holds single value as an element
# creating a dictionary
dict1 = {1: 'medium', 2: 'is', 3: 'best'}
dict2 = {1: 'single value', 2: {1:'another single value'}} # nested dictionary
print("Dictionary 1 values are",dict1," and Dictionary 2 values are",dict2)
Dictionary 1 values are {1: 'medium', 2: 'is', 3: 'best'} and Dictionary 2 values are {1: 'single value', 2: {1: 'another single value'}}
# adding elements to dictionary
dict3 = dict()
dict3[0] = "medium"
dict3[1] = "best"
print(dict3)
{0: 'medium', 1: 'best'}
# accessing dictionary
print(dict3[0]) # using index
medium
# removing elements from dictionary
dict1.clear()
print(dict1)
{}
# getting specific item using get and all the items in dictionary using items
print(dict2.get(1))
print(dict2.items())
print(dict1.keys()) # getting keys
single value
dict_items([(1, 'single value'), (2, {1: 'another single value'})])
dict_keys([])
# updating dictionary values
dict3.update({1:'bestest'})
print(dict3)
{0: 'medium', 1: 'bestest'}
# retriving values from dictionary
print(dict3.values())
dict_values(['medium', 'bestest'])
Array data structure in Python is a collection of same data type values at contiguous memory locations
# creating an array
import array as arr # importing array library for creating array
a = arr.array('i', [1, 2, 3]) # i here means integer type
print(a)
array('i', [1, 2, 3])
# updating the array
a.append(4)
a.insert(4,5)
print(a)
array('i', [1, 2, 3, 4, 5])
# removing array element
a.remove(5)
print(a)
array('i', [1, 2, 3, 4])
# slicing the array
b = arr.array('i',[1,2,3,4,5,6,7,8,9,10])
print("Sliced array b",b[4:9])
Sliced array b array('i', [5, 6, 7, 8, 9])
# searching the array
print(b.index(9))
8
Python Condition Programming
Discussing about various conditional programming concepts present in python such as OOPS, conditional statements and exception handling
# Loops
# while loop
i = 0
print("printing while loop")
while (i < 5):
print(i)
i = i +1
# for loop
print("printing for loop")
for i in range(0,5):
print(i)
# loop control statements
print("continue control statment")
for letter in "this_is_myarticle":
if letter == "e":
continue # continue returns control to the beginning of the loop
print("Letter currently,",letter)
print("break control statment")
for letter in "this_is_myarticle":
if letter == "e":
break # break stop the control of loop
print("Letter currently,",letter)
print("pass control statment")
for letter in "this_is_myarticle":
pass # empty loop kinda like place holder
print("Letter currently,",letter)
# loop with else
print("Loop with else statment")
for i in range(0,3):
print(i)
else:
print("if no break statment in loop else executes")
printing while loop
0
1
2
3
4
printing for loop
0
1
2
3
4
continue control statment
Letter currently, t
Letter currently, h
Letter currently, i
Letter currently, s
Letter currently, _
Letter currently, i
Letter currently, s
Letter currently, _
Letter currently, m
Letter currently, y
Letter currently, a
Letter currently, r
Letter currently, t
Letter currently, i
Letter currently, c
Letter currently, l
break control statment
Letter currently, t
Letter currently, h
Letter currently, i
Letter currently, s
Letter currently, _
Letter currently, i
Letter currently, s
Letter currently, _
Letter currently, m
Letter currently, y
Letter currently, a
Letter currently, r
Letter currently, t
Letter currently, i
Letter currently, c
Letter currently, l
pass control statment
Letter currently, e
Loop with else statment
0
1
2
if no break statment in loop else executes
# OOP's Object Oriented Programming uses classes and objects it provides real life entities like inheritance, polymorphisms, encapsulations and more.
# Class - collection of objects , it contains blueprint for objects throught which objects are made.
class Human:
pass # classes are made with class keyword, attributes are variables that belong to class, attributes are always public and can be accessed using dot operator.
# e.g Human.leftarm
# Object - entity with state and behaviour along with it
child_object = Human()
# self is similar to pointer in C++, the class method must have extra first parameter in method definition.
# __init__ is similar to constructor in C++ and Java and seen as soon as object of a class is instantiated.
class Human:
cattr = "mammal"
def __init__(self,name):
self.name = name
aditya = Human("Aditya") # object creation
# accessing class attributes
print(aditya.__class__.cattr)
# accessing class instance
print(aditya.name)
# Inheritance - using/inhertiing properities from another class
class Dog(object): # parent class
def __init__(self, name):
self.name = name
class Breed(Dog): # child class
def __init__(self, name):
Dog.__init__(self, name)
def detail(self):
print("Dog Breed is {}".format(self.name))
# creation of an object variable or an instance
a = Breed('German Shephard')
a.detail()
# Polymorphism - having many forms
class Dog:
def detail(self):
print("this is dog class method")
class Animal(Dog):
def detail(self):
print("this is animal class method")
obj_dog = Dog()
obj_animal = Animal()
print(obj_dog.detail(),obj_animal.detail())
# Encapsulation - it wraps data and methods in one unit
class Animal: # class is a example as it contains methods , variables etc
category = ""
def __init__(self,name):
self.name = name
class Insect(Animal):
def __init__(self):
Animal.__init__(self)
Animal.category = "insect"
print(self.category)
a = Animal("Beatle")
print(a.name)
mammal
Aditya
Dog Breed is German Shephard
this is dog class method
this is animal class method
None None
Beatle
# function - block of staments that return the specific task
# function definition
def fun(a): # def is used to define function and fun is function name and a is parameter
print("Number supplied to function fun is",a)
fun(10) # calling the function
def sum(a,b):
return a + b # return statment returns the value when the function is called
print(sum(3,2)) # this function returns sum of 2 numbers that are supplied
def pname(name=None):
print("Name provided to function pname is",name)
pname() # calling the function with default parameters
# variable length arguments *argv - non keyword attributes and **kwargs - keyword attributes with key and value pair
def non_keypair(*argv):
for variable in argv:
print(variable)
def keypair(**kwargs):
for key,value in kwargs.items():
print(key,value)
non_keypair("hello","world") # non keypair function
keypair(f1="hello",f2="world") # keypair function
# pass by reference
def xlist(x):
x[2] = 45
l1 = [1,2,3,4]
xlist(l1)
print(l1)
# pass by value
def ylist(y):
y = [10,20,30]
ylist(l1)
print(l1)
# anonymous function fnctions that are not named
square = lambda x : x*x
print(square(2))
Number supplied to function fun is 10
5
Name provided to function pname is None
hello
world
f1 hello
f2 world
[1, 2, 45, 4]
[1, 2, 45, 4]
4
Exception handling in python is managed using three keywords try, except and finally. Exceptions are instances that are raised when some internal event occurs which changes the flow of program.
# exception example
value = 10
if value > 9:
print("its bigger than 9")
Execution Error
IndentationError: expected an indented block (951488480.py, line 6)
# handling the the below error of divison by zero
value = 10
try:
if value / 0:
print("dividing value varriable by zero")
except:
print("error occured")
# can specify specific error like
try:
if value / 0:
print("dividing value varriable by zero")
except ZeroDivisionError:
print("ZeroDivisionError occured")
finally:
print("finally executes regarless or whether exception occured or not")
# Raising own exception with raise keyword
try:
raise("Raising exception")
print("Try block")
except:
print("Exception occured")
error occured
ZeroDivisionError occured
finally executes regarless or whether exception occured or not
Exception occured
Python Data Manipulation
Data manipulation techniques and ways present in Python for Data Science work namely data preprocessing, data transformation and data visualization.
Data preprocessing is the task of converting data from a given form to a much useful and consumable form.
# performing array operation using numpy
import numpy as np
a1 = np.arange(4,dtype=np.float_).reshape(2,2)
print(a1)
a2 = np.array([10,20])
print(a2)
# array operations - add subtract multiply divide
print(np.add(a1,a2))
print(np.subtract(a1,a2))
print(np.multiply(a1,a2))
print(np.divide(a1,a2))
[[0. 1.]
[2. 3.]]
[10 20]
[[10. 21.]
[12. 23.]]
[[-10. -19.]
[ -8. -17.]]
[[ 0. 20.]
[20. 60.]]
[[0. 0.05]
[0.2 0.15]]
Data Transformation is process of converting data from one format or structure into another format or structure
# Pandas - This library will be used for transforming dataframes using ways such as slicing,indexing, manipualting and cleaning
import pandas as pd
employee_list = [
["Aditya",21,4500000],
["Annie",22,3400000],
["Aman",25,1200000]
]
df = pd.DataFrame(employee_list, columns = ['Name','Age','Salary'])
print(df)
# slicing columns in dataframe
print(df.iloc[:,0:2])
# adding new column derived from other column(s)
df["Tax"] = df["Salary"]*0.45
print(df)
# sorting dataframe
df.sort_values(by=['Tax'])
print(df)
# checking for missing values
print(df.isnull().sum())
# df = df.dropna() in case of nan values and missing values
print(df.isnull()) # returns True if null else false
# filling null value with values
df.fillna(0) # for filling with zero
df.fillna(method='pad') # filling previous value
df.fillna('Empty', inplace = True) # filling na value with "Empty" string value
# replacing NaN value in dataframe with 1 value
df.replace(to_replace = np.nan , value = "1")
# reading csv file in pandas is done using pd.read_csv("path_to_file")
# reading json file in pandas is done using pd.read_json("path_to_file")
Name Age Salary
0 Aditya 21 4500000
1 Annie 22 3400000
2 Aman 25 1200000
Name Age
0 Aditya 21
1 Annie 22
2 Aman 25
Name Age Salary Tax
0 Aditya 21 4500000 2025000.0
1 Annie 22 3400000 1530000.0
2 Aman 25 1200000 540000.0
Name Age Salary Tax
0 Aditya 21 4500000 2025000.0
1 Annie 22 3400000 1530000.0
2 Aman 25 1200000 540000.0
Name 0
Age 0
Salary 0
Tax 0
dtype: int64
Name Age Salary Tax
0 False False False False
1 False False False False
2 False False False False
Nameobject
Ageint64
0
Aditya
21
1
Annie
22
2
Aman
25
Data wrangling is the process of gathering, collecting, and transforming Raw data into another format for better understanding, decision-making, accessing, and analysis in less time.
It deals with issues such as data exploration, dealing with mining values, reshaping data, filtering data. All of the them are discussed above.
Data Visualization is the process of presenting data in the form of graphs or charts. It helps to understand large and complex amounts of data very easily. It allows the decision-makers to make decisions very efficiently and also allows them in identifying new trends and patterns very easily.
# data visualization using matplot library
import matplotlib.pyplot as plt
x = [10, 20]
y = [20, 25]
# plotting the data
plt.plot(x, y)
plt.title("Straight Line") # adding title
plt.ylabel("Y axis") # Y axis label
plt.xlabel("X axis") # X axis label
plt.show()
plt.plot(x, y)
# Adding title to the plot
plt.title("Linear graph", fontsize=20, color="Blue")
# Adding label on the y-axis
plt.ylabel('Y-Axis')
# Adding label on the x-axis
plt.xlabel('X-Axis')
# Setting the limit of y-axis
plt.ylim(0, 80)
# setting the labels of x-axis
plt.xticks(x, labels=["one", "two"])
# Adding legends
plt.legend(["GFG"])
plt.show()
from matplotlib.figure import Figure
fig = plt.figure(figsize = (5, 4))
# Adding the axes to the figure
ax = fig.add_axes([1, 1, 1, 1])
# plotting 1st dataset to the figure
ax1 = ax.plot(x, y)
# plotting 2nd dataset to the figure
ax2 = ax.plot(y, x)
# Setting Title
ax.set_title("Linear Graph")
# Setting Label
ax.set_xlabel("X-Axis")
ax.set_ylabel("Y-Axis")
# Adding Legend
ax.legend(labels = ('line 1', 'line 2'))
plt.show()
# sub plot in matplot
# Creating figure object
plt.figure()
# adding first subplot
plt.subplot(121)
plt.plot(x, y)
# adding second subplot
plt.subplot(122)
plt.plot(y, x)
axes1 = plt.subplot2grid (
(7, 1), (0, 0), rowspan = 2, colspan = 1)
axes2 = plt.subplot2grid (
(7, 1), (2, 0), rowspan = 2, colspan = 1)
# plotting the data
axes1.plot(x, y)
axes2.plot(y, x)
# various graphs in matplotlib
# line graph was already dicussed above
# Bar graph
import matplotlib.pyplot as plt
import pandas as pd
data = pd.read_csv('Data-Science-with-Python/tips.csv')
# initializing the data
x = data['day']
y = data['total_bill']
# plotting the data
plt.bar(x, y)
# Adding title to the plot
plt.title("Tips Dataset")
# Adding label on the y-axis
plt.ylabel('Total Bill')
# Adding label on the x-axis
plt.xlabel('Day')
plt.show()
# Histogram
import matplotlib.pyplot as plt
import pandas as pd
# Reading the tips.csv file
data = pd.read_csv('Data-Science-with-Python/tips.csv')
# initializing the data
x = data['total_bill']
# plotting the data
plt.hist(x)
# Adding title to the plot
plt.title("Tips Dataset")
# Adding label on the y-axis
plt.ylabel('Frequency')
# Adding label on the x-axis
plt.xlabel('Total Bill')
plt.show()
# scatter plot
import matplotlib.pyplot as plt
import pandas as pd
# Reading the tips.csv file
data = pd.read_csv('Data-Science-with-Python/tips.csv')
# initializing the data
x = data['day']
y = data['total_bill']
# plotting the data
plt.scatter(x, y)
# Adding title to the plot
plt.title("Tips Dataset")
# Adding label on the y-axis
plt.ylabel('Total Bill')
# Adding label on the x-axis
plt.xlabel('Day')
plt.show()
# pie chart
import matplotlib.pyplot as plt
import pandas as pd
# Reading the tips.csv file
data = pd.read_csv('Data-Science-with-Python/tips.csv')
# initializing the data
cars = ['AUDI', 'BMW', 'FORD',
'TESLA', 'JAGUAR',]
data = [23, 10, 35, 15, 12]
# plotting the data
plt.pie(data, labels=cars)
# Adding title to the plot
plt.title("Car data")
plt.show()
# to save use plt.savefig("output.jpg")
Data visualization using seaborn also as it offer more graphs and more attractive visuals
import seaborn as sns
sns.set( style = "white" )
# ploting a histogram and kde with binsize determined automatically
sns.distplot(data, kde = True, color = "g")
/tmp/ipykernel_685/1187558920.py:6: UserWarning:
`distplot` is a deprecated function and will be removed in seaborn v0.14.0.
Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).
For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751
sns.distplot(data, kde = True, color = "g")
# line plot
import pandas as pd
data = pd.read_csv("Data-Science-with-Python/nba.csv")
# plotting lineplot
sns.lineplot( data['Age'])
sns.scatterplot(data['Age'])
sns.boxplot( data['Age'] )
sns.violinplot(data['Age'])
sns.swarmplot(x = data["Age"])
sns.barplot(x =data["Age"])
sns.pointplot(x = "Age", y = "Weight", data = data)
sns.countplot(data["Age"])
Statistics with Python
# Measure of central tendency using pandas
# checking on the column Age
print(data['Age'].mean(),data['Age'].median(), data['Age'].mode(),data['Age'].std,data['Age'].var())
26.938730853391686 26.0 0 24.0
dtype: float64 <bound method NDFrame._add_numeric_operations.<locals>.std of 0 25.0
1 25.0
2 27.0
3 22.0
4 29.0
...
453 26.0
454 24.0
455 26.0
456 26.0
457 NaN
Name: Age, Length: 458, dtype: float64> 19.395360666436332
data.describe() # describing statistically the dataset
Numberfloat64
Agefloat64
count
457.0
457.0
mean
17.678336980306344
26.938730853391686
std
15.966090405679644
4.404016424405833
min
0.0
19.0
25%
5.0
24.0
50%
13.0
26.0
75%
25.0
30.0
max
99.0
40.0