Extracting Song Data From Spotify using a Web API

This is a tutorial on how audio data can be extracted using Spotify's Web API (Application Program Interface) which is based on simple REST principles. The Spotify Web API endpoints return JSON metadata containing data about music artists, albums, and tracks. Furthermore, in-depth statistics on songs such as features describing the "feel" of audio in terms of its "liveness", "acoustics" and "energy" can be retrieved. Also, it provides access to user-related data such as playlists, and music that the user saves in the Your Music library. In this tutorial, we will focus on extracting audio data on songs uploaded or created by an artist.

JSON stands for JavaScript Object Notation developed by Douglas Crockford in the 2000s.

Libraries

There are two Python libraries or packages available for the extraction of audio data from Spotify, namely plamere/spotipy and tekore. These are wrapper classes that are embedded with HTTP request methods such as GET and POST which are used to access data via Spotify's host URL: api.spotify.com. We will use the "Spotipy" library for this tutorial.

import os # for operating system dependant functionalities import numpy as np # for mathematical operations import pandas as pd # for tabulating data from sklearn import preprocessing # for data preprocessing / Machine Learning Task import matplotlib.pyplot as plt # for visualization import seaborn as sns # for visualization import spotipy # spotipy wrapper package from spotipy.oauth2 import SpotifyClientCredentials # for authentication CLIENT_ID = os.environ["CLIENT_ID"] # Spotify ID is stored CLIENT_SECRET = os.environ["CLIENT_SECRET"] # Spotify Secret Key is stored # creating an object sp = spotipy.Spotify(auth_manager=SpotifyClientCredentials(client_id=CLIENT_ID, client_secret=CLIENT_SECRET))

It is not adviceable to reveal your Spotify ID and Key if you intend to publish your Python Code or Notebooks (here is a video explaining how to hide them). There are two ways to extract data from Spotify. It's either with or without user authentication. The latter is used when your application does not need to authenticate users, however, you would still need to get a Spotify ID and Key to connect to the Spotify's servers. To get started, head to Spotify, create an app, and add your new ID and SECRET to your environment:

Data Extraction

Let's begin with a basic search of an artist name which will be passed as a string. We will search for the name "Sarkodie" and NOT "sarkodie".

# creating a search query function def search_spotify(q:str, limit:int): global search_results # accessible globally in this notebook search_results = sp.search(q=q, limit=limit) # limit: 0 < limit <= 50 [print(idx, track['name']) for idx, track in enumerate(search_results['tracks']['items'])] print('\n Total number of items available to return: {}'.format(search_results['tracks']['total'])) # This will return 25 albulms/tracks associated with the word "Sarkodie" # These could be albums, tracks or features with "Sarkodie" search_spotify('Sarkodie', 25)

The function takes two arguements, a string and an integer, that are passed on to the search object in line three of the function. The forth line is a list comprehension that prints an enumerated list of song names returned in the JSON file. The fifth line returns the total number of items available to return.

This search returned some popular hits of Sarkodie and other songs he might have been featured on. Below is a structure of the returned JSON file:

The root node is called tracks and within tracks there are 7 parent nodes namely href, items, limit, next, offset, previous, and total. The href parent node contains the URL of the API host address. Inside items is where the list of albums are returned. Album could be type album, single, or compilation. Limit is an integer; the number of items returned in each set (the maximum limit allowed for each set of results is 50). If you have more results than the limit specified, the next node will provide a URL that has the subsequent set of list to retrieve. The offset is an integer that sets the index for the first returned results; this is used in getting the next pages/lsit of songs. The total node returns the total number of search results that can be querried.

At this point we wish to extract song data from a particular artisit. What we need is the artist's Spotify ID. There are three ways to retrieve this information:

using a query search engine such as Google i.e. search the term "spotify sarkodie" and first link https://open.spotify.com/artist/01DTVE3KmoPogPZaOvMqO8 will send you to Sarkodie's Spotify profile. The link has the Spotify ID embedded in there which is 01DTVE3KmoPogPZaOvMqO8 [1]

from the JSON file that was returned by looping through the children nodes.

or simply asking artisit. LOL, that's if his Spotify ID is private!

Some of the tracks/albums may have more than one artist and we want to be sure we are getting the right Spotify ID of the artist in question. In the JSON file there two places we can retrieve the Spotify ID of an artist: In the spotify and uri node. The spotify node has the Spotify profile page of an artist, similar to the link above in [1]. The uri which stands for Universal Record Identifier, stores a unique identifier for every artist on spotify; it looks like this: "spotify:artist:spotify_id". The Spotify ID will be in the third token where is says "spotify_id".

# items isA list, so therefore iterate over list with a numpy array for x in np.arange(len(search_results['tracks']['items'])): # artist isA list as well, so loop over the list with an enumerator for idx, name in enumerate(search_results['tracks']['items'][x]['album']['artists']): if 'Sarkodie' in name['name']: print("{}'s URI is {}".format(name['name'], name['uri'])) break # break out of inner loop if condition is met if 'Sarkodie' in name['name']: break # break out of outer loop by checking if condition is met

Sakordie's Spotify ID is "01DTVE3KmoPogPZaOvMqO8". Now lets get Sarkodie's tracks/albums on Spotify.

sarkodie_uri ='spotify:artist:01DTVE3KmoPogPZaOvMqO8' album_types =["album", "single","compilation"] album_results = sp.artist_albums(sarkodie_uri, album_type='album') albums = album_results['items'] while album_results['next']: album_results = sp.next(album_results) albums.extend(album_results['items']) print('Total number of albums: {}'.format(album_results['total'])) tracks = 0 for album in albums: tracks = tracks + album['total_tracks'] print('Album: {}, Total Tracks: {}, Release Date: {}, ID: {}'\ .format(album['name'], album['total_tracks'], album['release_date'], album['id'])) print(f'Total number of tracks: {tracks}')

Sarkodie has 7 ablums on Spotify with a grand total of 119 tracks. For this tutorial we are going to extract the tracks for Sarkodies latest album JAMZ. But before we proceed we need the album's ID which is: 4N96XJi7wu1B0ACzCgPLLc

# Let's get some stats on Sarkodie artist_stats = sp.artist(sarkodie_uri) # artist object method print('{} currently has {:,}K followers and a popularity of {} out of 100.'.format(artist_stats['name'], artist_stats['followers']['total'], artist_stats['popularity'])) print('This artist performs the following genres {}'.format(artist_stats['genres']))

country_codes= ['AD', 'AE', 'AG', 'AL', 'AM', 'AO', 'AR', 'AT', 'AU', 'AZ', 'BA', 'BB', 'BD', 'BE', 'BF', 'BG', 'BH', 'BI', 'BJ', 'BN', 'BO', 'BR', 'BS', 'BT', 'BW', 'BY', 'BZ', 'CA', 'CD', 'CG', 'CH', 'CI', 'CL', 'CM', 'CO', 'CR', 'CV', 'CW', 'CY', 'CZ', 'DE', 'DJ', 'DK', 'DM', 'DO', 'DZ', 'EC', 'EE', 'EG', 'ES', 'FI', 'FJ', 'FM', 'FR', 'GA', 'GB', 'GD', 'GE', 'GH', 'GM', 'GN', 'GQ', 'GR', 'GT', 'GW', 'GY', 'HK', 'HN', 'HR', 'HT', 'HU', 'ID', 'IE', 'IL', 'IN', 'IQ', 'IS', 'IT', 'JM', 'JO', 'JP', 'KE', 'KG', 'KH', 'KI', 'KM', 'KN', 'KR', 'KW', 'KZ', 'LA', 'LB', 'LC', 'LI', 'LK', 'LR', 'LS', 'LT', 'LU', 'LV', 'LY', 'MA', 'MC', 'MD', 'ME', 'MG', 'MH', 'MK', 'ML', 'MN', 'MO', 'MR', 'MT', 'MU', 'MV', 'MW', 'MX', 'MY', 'MZ', 'NA', 'NE', 'NG', 'NI', 'NL', 'NO', 'NP', 'NR', 'NZ', 'OM', 'PA', 'PE', 'PG', 'PH', 'PK', 'PL', 'PS', 'PT', 'PW', 'PY', 'QA', 'RO', 'RS', 'RW', 'SA', 'SB', 'SC', 'SE', 'SG', 'SI', 'SK', 'SL', 'SM', 'SN', 'SR', 'ST', 'SV', 'SZ', 'TD', 'TG', 'TH', 'TJ', 'TL', 'TN', 'TO', 'TR', 'TT', 'TV', 'TW', 'TZ', 'UA', 'UG', 'US', 'UY', 'UZ', 'VC', 'VE', 'VN', 'VU', 'WS', 'XK', 'ZA', 'ZM', 'ZW'] # A list of countries where Spotify is available; can be used for an in-dept analysis JAMZ_id = '4N96XJi7wu1B0ACzCgPLLc' track_results = sp.album_tracks(JAMZ_id, market=None, limit=50) # album tracks object method tracks = track_results['items'] while track_results['next']: track_results = sp.next(results_tracks) tracks.extend(track_results['items']) track_ids = [] # store all the track ids asA list track_names = [] # store all the names of the tracks asA list for track in tracks: print('{} ID: {}'.format(track['name'], track['id'])) track_names.append(track['name']) track_ids.append(track['id'])

Exploratory Data Analysis (EDA)

# Let's now get some metrics from these tracks such as acousticness, danceability, # duration, liveness, etc. to explore the data visually track_metrics = sp.audio_features(tracks=track_ids) # get metrics of tracks track_popularity = sp.tracks(track_ids)['tracks'] # get popularity of tracks track_popularity_list = [] # store the popularity of each track [track_popularity_list.append(pop['popularity']) for pop in track_popularity] track_metrics_df = pd.DataFrame.from_dict(track_metrics) # transform the JSON file into a tabular data track_metrics_df.info() # display the metadata of the table

# Let's keep only the numeric columns of the DataFrame for analysis track_metrics_df = track_metrics_df.select_dtypes(include='number') # Now lets add a column with the track names track_metrics_df['popularity'] = track_popularity_list columns_for_normalized_df = track_metrics_df.columns track_metrics_df['name'] = track_names # Now lets look at the DataFrame track_metrics_df

The following list explains the metrics that come with each track.

Instrumentalness: This value represents the amount of vocals in the song. The closer it is to 1.0, the more instrumental the song is.

Acousticness: This value describes how acoustic a song is. A score of 1.0 means the song is most likely to be an acoustic one. Liveness: This value describes the probability that the song was recorded with a live audience. According to the official documentation “a value above 0.8 provides strong likelihood that the track is live”.

Speechiness: “Speechiness detects the presence of spoken words in a track”. If the speechiness of a song is above 0.66, it is probably made of spoken words, a score between 0.33 and 0.66 is a song that may contain both music and words, and a score below 0.33 means the song does not have any speech.

Energy: “(energy) represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy”.

Danceability: “Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable”.

Valence: “A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry)”.

track_metrics_df.describe()

Visuals

Let's get some statistical insights into our data. For our data analysis, we will take a deep dive into learning how to show patterns, correlations, and visualize our data to the common eye. Creating visually appealing statistics about our data is important because the DataFrame is a bit hard to understand, especially in the general scope. Through graphs and plots, we will be able to better understand our data and get a better overview of it.

# plot a heat map plt.figure(figsize=(16, 8)) sns.set(style="whitegrid") corr = track_metrics_df.corr() sns.heatmap(corr,annot=True, cmap="YlGnBu")

The highest correlating metrics are the duration and instrumentalness with a score of 0.81. The song titled She Bad (feat. Oxlade) has the longest duration of about 5 mins and has higher instrumentalness than the other songs in JAMZ.

The next highest correlation is between danceability and valence with a score of 0.69. 7 of the songs are all over the mean danceability of 0.82 whereas 5 of the songs are in above the mean valence. Cougar (feat. Lojay) is the most danceable song according to Spotify followed by Country Side (feat. Black Sherif). In the valence category Labadi (feat. King Promise) provides listeners with more positivity, followed by Cougar (feat. Lojay) and then Country Side (feat. Black Sherif).

Other observations with high correlations are loudness & speechness with 0.66, liveness and tempo with 0.63, and then energy and tempo with 0.62.

Spotify has its own way of calculating a track's popularity. Usually popularity should correlate with the number of streams, however, Spotify generates the popularity metric on how recent the streams of the song are, in order words its streaming frequency. The most popular song in JAMZ are She Bad (feat. Oxlade), Country Side (feat. Black Sherif) and Better Days (feat. BNXN fka Buju). In this visual, popularity correlates positively with the mode, which is the modality (major or minor) of the song. Popularity also correlates with duration_ms, and liveness.

With these findings I decided to create a formula to rate the songs:

new_popularity = danceability + energy + loudness + tempo

Supposing you are at a party, preferably you would want to dance to loud, energetic music with a bit of tempo, thus, I created the mathematical function above.

#normalize the data for uniformity x = track_metrics_df.select_dtypes(include='number').values # transform the DataFrame into a Series (array) min_max_scaler = preprocessing.MinMaxScaler() # define a Min-Max normalizer object x_scaled = min_max_scaler.fit_transform(x) # fit the data using the object #transform the normalized array back into a DataFrame track_metrics_df_norm = pd.DataFrame(x_scaled, columns=columns_for_normalized_df) # To have uniform calculation we used the normalized table track_metrics_df_norm['new_popularity'] = track_metrics_df_norm['danceability'] + track_metrics_df_norm['energy'] + track_metrics_df_norm['loudness'] + track_metrics_df_norm['tempo'] track_metrics_df_norm['name'] = track_names #track_metrics_df_norm['popularity'] = track_popularity_list plt.figure(figsize=(16, 8)) sns.set(style="whitegrid") corr = track_metrics_df_norm.corr() sns.heatmap(corr,annot=True, cmap="YlGnBu")

In the heat map above the calculated popularity is positively correlated with danceability, energy, liveness, and tempo.

Voilà, with the calculated popularity Country Side (feat. Black Sherif) receives the highest popularity followed by Labadi (feat. King Promise) and then Better Days (feat. BNXN fka Buju). And by the way this is according to my Dance Formula.

I hope you enjoy going through this tutorial. If you have any questions or suggestions follow me on Twitter (@kingKwabs) and send me text me. This analysis is as of November 12, 2022 and the data retrieved may be not reflect the current trends of the artist used for this analysis.

References

String Formats: Link

Python String Formatting: Link

Spotipy Python Library: Link

JSON Validator: Link

Music attributes and its effect on popularity: Link