Voice recognition for ES 156 (Spring 2021)

Acknowledgments: the dataset was downloaded from https://github.com/Jakobovski/free-spoken-digit-dataset, all credits to that repo. We also acknowledge Prof. Alejandro Ribeiro (UPenn) whose lecture notes inspired this exercise.

Data cleanup

First we start by importing a few packages.

import numpy as np import numpy.random as random from numpy.fft import fft from scipy.io import wavfile import matplotlib.pyplot as plt import seaborn as sns import os %matplotlib inline sns.set() sns.set(font_scale=1.5)

Next we load the data. By changing the digits in the list digit, you can import different spoken digits from the dataset. The dataset consists of .wav files -- you can download it and give it a listen!

The recordings are all sampled at 8KHz. The recordings, however, have different lengths due to different recording durations. The maximum vector length will determine the length of the FFT (shorter vectors will be padded with zeros). This is given by the variable `.

The dictionary signals will store the recorded spoken digits. So signals[d] returns a list of vectors that contains recordings for digit d.

data_dir = './recordings/' # determine digits of interest (0 to 9) 0,1,2,3,4,5,6,7,8,9 digits = [0,1,2,3,4,5,6,7,8,9] # change here to load more digits # dictionary that will store our values signals = {d:[] for d in digits} file_names = {d:[] for d in digits} # import files for filename in os.listdir(data_dir): # iterate over digits for d in digits: if filename.startswith(str(d)+'_'): wav = wavfile.read(data_dir+filename)[1] if len(wav.shape)<2: signals[d].append(wav) file_names[d].append(filename) # find maximum of vector length N = max([len(v) for d in digits for v in signals[d]]) #pad vectors with 0's if len less than N for d in digits: for i in range(len(signals[d])): if len(signals[d][i]) < N: signals[d][i] = np.pad(signals[d][i], (0, N-len(signals[d][i])), 'constant')

Classification

In order to perform classification, the dataset has to be split in "training" and "testing". The training dataset will be used to determine the average spectrum used in the next classification tasks. The test dataset will be used to evaluate the performance of our method. We create a random split 80/20 split.

# next we split our dataset in train and test # we will use a 80/20 random split. # create train/test split ix = np.arange(100) random.shuffle(ix) # select train entries ix_train = ix[:80] #select test entries ix_test = ix[80:]

Next we compute the average spectral magnitude of the DFT of each digit using the FFT algorithm.

Let $x{i,d}n$ be the discrete signal corresponding to the $i$th instance of digit $d$ in the dataset. For example, each of $x{1,1}n,x{2,1}n,\dots,x{m,1}n$ is the signal corresponding to one of the $m$ instances of digit 1.

We can compute the transform (via the FFT) of each of these signals: $$x{i,d}n\leftrightarrow X{i,d}k. $$

We define the average spectral magnitude of digit $d$ as the average over all instances of that digit in the dataset: $$\bar{X}{d}k \triangleq \sum{i\in \mathrm{instances}}\frac{\left|X_{i,d}k\right|}{\mathrm{number~of~instances}}.$$

We normalize this quantity to create the normalized average spectral magnitude of digit $d$:

$$\tilde{X}{d}k\triangleq\frac{\bar{X}{d}k}{\sqrt{\sum{k'}\left|\bar{X}{d}k'\right|^2}}.$$

Later we will use this as a "signature" for each digit. A new instance of a spoken digit will be classified as digit $d$ if the normalized spectral magnitude of that new signal is similar to the normalized average spectral magnitude of the $d$th digit $\tilde{X}_{d}k$.

Question 1:

Compute $\tilde{X}_{d}k$ for all digits and store them in the dictionary mean_transforms below.

# next we compute the average spectrum of each spoken digit in the training set. # we will consider a window up to 1.5 KHz # sampling rate is 8kHz Ts = 1.0/8000 ix_cut = int(np.ceil(1500*Ts*N)) # initialize dictionary for storing transforms transforms = {} # initialize dictionary for storing mean transforms mean_transforms = {} # compute mean transform of each digit and in the training set. # Make sure to only keep the spectrum up to 1.5kHz

# Code Solution to Q1 Here transforms = {d:[] for d in digits} #for each digit stores the list of absolute values of FFT of data file. average = {d:[] for d in digits} #average spectral magnitude for digit d mean_transforms = {d:[] for d in digits} #normalized average for d in digits: transforms[d] = [[abs(fft(signals[d][i],norm='ortho',n=N))] for i in ix_train] #For instances i in ix_train take fft average[d] = np.mean(transforms[d], axis=0) #element wise averages of t mean_transforms[d] = average[d]/np.linalg.norm(average[d]) #Normalize average[d] into mean transforms

Question 2: Plot the average spectral magnitude of each digit

You should create a plot that averages the of the magnitude of the FFT transforms of all signals (i.e., recordings) of a given digit.

# In this next part, plot the average spectral magnitude of each digit.

# Code Solution to Q2 here for d in digits: y = mean_transforms[d][0] x = np.linspace(0.0, 1.0/(2.0*Ts), int(N)) plt.plot(x,y) plt.xlabel('Frequency (Hz)') plt.ylabel('Amplitude') plt.grid() plt.show()

If you selected digits 1 and 2, you can see that there is a very clear distinction between both digits. We create next the mean classifier for distinguishing between spoken digits. For a given signal $xn\leftrightarrow Xk$, the mean classifier computes the inner product

$$p\left(X,\tilde{X}d\right)\triangleq \sum{k=1}^N \left|Xk\right|\left|\tilde{X}_dk\right|, $$

where $$\left|\tilde{X}_dk\right|$$ is the mean spectral magnitude for digit $d$. We refer to this quantity as the similarity between $X$ and $\tilde{X}_d$. It quantifies how similar the magnitude of the spectrum of $X$ is with the average spectrum of digit $d$. The mean classifier then outputs as the class of $xn$ the digit with the highest similarity.

Question 3.a: Write a classifier function that returns the digit for which $xn$ has the highest (spectral) similarity.

# classifier function # receives a vector, computes the product with average digits, and returns the max inner product # Input: sample x (vector) def mean_classifier(x): similar = {d:[] for d in digits} #similiarities for each digit d xft = np.abs(fft(x)) #magnitude of FFT for signal x for d in digits: similar[d] = np.inner(xft/np.linalg.norm(xft), mean_transforms[d]) #inner product of normalized xft and mean transforms most_sim = max(similar.values()) #Most similar for d in digits: #Assign d if similar[d] == most_sim: return d

# Code Q3a Here

Question 3.b: Apply this classifier to the test set and see how it performs.

# Write anser for Q3b here

# Code 3b Here percent = {d:[] for d in digits} for d in digits: total = 0 correct = 0 for i in ix_test: #Each instance in testing split if mean_classifier(signals[d][i]) == d: #If signal belong to the right digit correct += 1 #classified in correct bin total += 1 percent[d] = correct/(total) #Percentage correct for that digit print(percent)

Question 4: How does the accuracy generalize when you classify more digits?

# Write answer for Q4 here # Generally, the accuracy of the classifer decreases when # we increase the number of digits to 10 because # you have more bins to classify each signal in. Therefore # more mistakes can be made in classifying the # signal since there are more chances that the signals being # tested are incidentally similar to another # digit.

Write Q4 Solution here

# Code Q4 here

Question 5 (Bonus + 5 points): Can you design a more accurate classifer than the average classifier given above?

# Write your answer here