Building a Wine Recommendation System With a Twist
Wines seem to lend themselves very well to data science projects for a number of reasons: availability of data, number of features available per wine and general love for the stuff being a few of the main driving factors. As such, there are no shortages of brilliant wine based projects on display, and for this project I wanted to take a slightly different approach towards creating a recommendation model. The approach I landed on was to build an NLP model which calculates similarity between wines based on the reviews of expert sommeliers.
Summary
Imports and Data Cleaning
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 150930 entries, 0 to 150929
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 150930 non-null int64
1 country 150925 non-null object
2 description 150930 non-null object
3 designation 105195 non-null object
4 points 150930 non-null int64
5 price 137235 non-null float64
6 province 150925 non-null object
7 region_1 125870 non-null object
8 region_2 60953 non-null object
9 variety 150930 non-null object
10 winery 150930 non-null object
dtypes: float64(1), int64(2), object(8)
memory usage: 12.7+ MB
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 150930 entries, 0 to 150929
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 country 150925 non-null object
1 description 150930 non-null object
2 designation 105195 non-null object
3 province 150925 non-null object
4 region_1 125870 non-null object
5 variety 150930 non-null object
6 winery 150930 non-null object
dtypes: object(7)
memory usage: 8.1+ MB
Missing Data
Now the predictor matrix has been created - let's tackle the missing data. For the first iteration of this recommender system, I will drop observations with missing values across the board instead of being more selective. This cuts the available data in half. Further iterations of this project should try a different, less brutal, approach towards missing data.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 85614 entries, 0 to 85613
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 index 85614 non-null int64
1 country 85614 non-null object
2 description 85614 non-null object
3 designation 85614 non-null object
4 province 85614 non-null object
5 region_1 85614 non-null object
6 variety 85614 non-null object
7 winery 85614 non-null object
dtypes: int64(1), object(7)
memory usage: 5.2+ MB
Feature Engineering Unique Names for Wines
Let's create a more detailed name for each wine by combining 'winery' and 'designation'.
Removing Duplicate Values
In order for the end user to recieve recommendations, using this model and approach, there needs to be a unique name for each wine. With so many duplicates this becomes tricky. For now I will drop duplicated wine values, which significantly reduces the volume of data but let's the project proceed. There are certainly much better ways to proceed here, but for the sake of moving the project on I am taking the quicker route here.
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 33549 entries, 0 to 33548
Data columns (total 8 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 country 33549 non-null object
1 description 33549 non-null object
2 designation 33549 non-null object
3 province 33549 non-null object
4 region_1 33549 non-null object
5 variety 33549 non-null object
6 winery 33549 non-null object
7 name 33549 non-null object
dtypes: object(8)
memory usage: 2.0+ MB
Reducing Scope of Data
At this point in the project I had successfully created the recommendation function, however, when I tried to deploy the model using Streamlit I recieved endless connection timeout errors. After many, many hours of digging and researching I came to the conclusion the issue must lie with the connection to the AWS s3 bucket hosting my model.pkl file and predictors.csv. The files were simply too large to be loaded into memory and cached on each running of the script compiled for deployment.
The solution was to reduce the scope of the project to focus on the wine of one country instead of all countires, hence reducing the size of the files involved. As I am a big fan of Italian wines I decided to put this bias to good use and make this an Italian wine recommendation project.
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6605 entries, 0 to 6604
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 index 6605 non-null int64
1 country 6605 non-null object
2 description 6605 non-null object
3 designation 6605 non-null object
4 province 6605 non-null object
5 region_1 6605 non-null object
6 variety 6605 non-null object
7 winery 6605 non-null object
8 name 6605 non-null object
dtypes: int64(1), object(8)
memory usage: 464.5+ KB
Creating a Search for Wine Feature
One of the features I wanted to add to the deployed project was the ability to click a button and immediately search for the wine being recommended. Having looked around for a wine website with a predictable URL syntax for searches, I landed on www.wine-searcher.com who simply append each search term to the URL between + symbols, alongside the name of the country. Knowing this I could create a search URL for each wine in the dataset.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
"""Entry point for launching an IPython kernel.
/shared-libs/python3.7/py/lib/python3.7/site-packages/pandas/core/frame.py:4315: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
errors=errors,
Vectorizing With Tf-idf
Time to turn the sommelier reviews into vectors using Tf-idf in order to proceed with the model. The parameters were chosen after a few rounds of trial and error - I had a feeling the ngram_range would work well set to 2 or 3 as some of the descriptive language being used in the descriptions were bigrams and trigrams ("sweet berry", "forest floor".etc). The regex pattern chosen was very much trial and error, using a description as an example and plugging it into regexr.com.
Calculating Similarity Scores
The Recommender
The recommender function works by taking the sigmoid_kernel scores and mapping them against the index pandas series, which is itself conceived by taking the index of the predictor matrix and the name values for each wine. The result is each wine is given a similary score to the other wines in the predictor matrix. The series is then sorted and the function returns the 3 wines with the highest similarity score to the wine passed in to the function.
Exporting the Model
Adapting for Streamlit
Adapting the above code for streamlit was a case of spending a few days reading through documentation and familiarising myself with the decorators and streamlit syntax. Once I had accomplished this, the next step was to move this code out of a notebook format and rewriting the code, using the streamlit decorators, in order to allow for user input and loading of model files from an s3 bucket. I highly recommend Streamlit for anyone looking to deploy their models - very user friendly syntax and intuitive to understand!