Mining Football Data: A Web-Scraping and Exploratory Analysis Project

English Premier League Data Scraping and Analysis

The English Premier League is renowned as the most popular football league globally, with its widespread media appeal and fast-paced, high-intensity football style. Analysts can relish the wealth of data generated throughout an entire season. In this project, we'll scrape goal-scoring statistics for the recently concluded 2022/23 season from the Premier League's official website and conduct exploratory data analysis (EDA).

Data Source

The official website of the English Premier League, https://www.premierleague.com, houses fascinating player statistics data that can be analyzed for intriguing insights. Although the website does not have a native API, we can still obtain the data directly from the HTML pages while considering the source data table's pagination.

Web Scraping

To overcome this challenge, we'll employ web scraping using a combination of Selenium and automation through Nerodia/Watir, while parsing the data with Pandas.

Web scraping is an automated method for extracting content and data from a website. Generally, websites contain unstructured data, but web scraping extracts and processes it, making it structured.

Now, let's dive into the workflow!

First, we define a list of URLs, each containing a specific player stat. We then create a function called "create_df" that uses Selenium to load the HTML table from the URL and convert it into a Pandas data frame. The function also performs data preprocessing, such as renaming and dropping unnecessary columns.

import time import pandas as pd from nerodia.browser import Browser from selenium.webdriver.chrome.options import Options options = Options() options.add_argument('--headless') options.add_argument('--no-sandbox') browser = Browser('chrome', options=options) tags = ['goals', 'total_pass', 'touches', 'total_scoring_att', 'big_chance_missed', 'appearances', 'total_offside', 'total_cross', 'total_through_ball','goal_assist'] se = '?se=489' urls = [f'https://www.premierleague.com/stats/top/players/{tag}{se}' for tag in tags] def create_df(url, stat_col): def read_table_page(): time.sleep(3) # time delay for data reload # Create dataframe by reading from html table return pd.read_html(browser.html)[0] browser.goto(url) df = read_table_page() # Iterate through pages, appending data to the dataframe while not browser.div(class_name=['paginationBtn', 'paginationNextContainer', 'inactive']).exists: # Fire onClick event on page next element browser.div(class_name=['paginationBtn', 'paginationNextContainer']).fire_event('onClick') # Append the table from this page with the existing dataframe df = pd.concat([df, read_table_page()], ignore_index=True) # Replace append with pd.concat and ignore_index=True # Data preprocessing df = df.rename(columns={"Stat": stat_col}) df = df.iloc[:, :-1] # Drop last "nan" column df = df.drop(columns='Rank') # Drop "rank" column return df goals_df = create_df(urls[0], 'Goals') total_pass_df = create_df(urls[1], 'Total Pass') touches_df = create_df(urls[2], 'Touches') total_scoring_att_df = create_df(urls[3], 'Shots') big_chance_missed_df = create_df(urls[4], 'Big Chance Missed') appearances_df = create_df(urls[5], 'Appearances') total_offside_df = create_df(urls[6], 'Total Offside') total_cross_df = create_df(urls[7], 'Total Cross') total_through_ball_df = create_df(urls[8], 'Total Through Ball') assists_df = create_df(urls[9],'Assists') browser.quit()

After, we call the "create_df" function for each URL in the list of URLs, passing in the URL and the corresponding stat column name. This creates a data frame for each player's stat.

import numpy as np dataframes = [ goals_df, total_pass_df, touches_df, total_scoring_att_df, big_chance_missed_df, appearances_df, total_offside_df, total_cross_df, total_through_ball_df, assists_df ] # Merge all DataFrames on the 'Player', 'Nationality', and 'Club' columns full_df = dataframes[0] for df in dataframes[1:]: full_df = pd.merge(full_df, df, on=['Player', 'Nationality', 'Club'], how='outer') # Display the full DataFrame full_df

Data Processing

The presence of a "-" value in the Clubs column may indicate that a player has been transferred, loaned to another club, or is currently injured.

We observe that the data contains NaN values for players who haven't registered any goals, passes, touches, shots, etc. To address this, we will replace all NaN values with 0.

# Remove all the NaN column with 0 full_df=full_df.fillna(0) full_df

Exploratory Data Analysis (EDA)

Let’s first look at the distribution of goals scored.

#Histrogram of Goals Scored import matplotlib.pyplot as plt import seaborn as sns plt.figure(figsize=(12,7)) g=sns.histplot(full_df['Goals']) plt.title("Distribution of Goals Scored") plt.show()

Only 5 players scored more than 15 goals in the season; with Manchester City's Erling Haaland and Tottenham Hotspur’s Harry Kane joint top scorers with 35 and 26 goals.

The distribution of goal scorers per club is illustrated in the boxplot below:

#Distribution of goals scored per club import plotly.express as px fig=px.box(full_df,y='Goals',x='Club',title='Goal Scoring Distribution Per Club',template='plotly_dark',hover_data=["Player"],color='Club') fig.show()

It is evident from the data that only two clubs, Manchester City and Arsenal, had upper fence values of 10 in terms of goals scored, with one outlier scoring 35 goals and two outliers scoring 15 and 13 goals, respectively. These clubs were the top performers in terms of goals scored in the 2022/2023 Premier League season. It is worth noting that all clubs in the Premier League had players who were considered outliers in terms of goal-scoring for the season.

Analyzing Variable Correlations

To examine the relationships between various metrics and their connection to goal-scoring, as well as their interdependence, a correlation matrix can be utilized. While correlation does not necessarily indicate causation, it provides a useful measure of the strength of the association between variables.

#Calculate pearson correlation corr = full_df.corr() plt.figure(figsize=(12,7)) # Generate a mask for the upper triangle mask = np.triu(np.ones_like(corr, dtype=bool)) corrPlot=sns.heatmap(corr,mask=mask,cmap='jet',annot=True) sns.set(style="dark") plt.title("Correlation Matrix for Player Stats") plt.show()

From the correlation matrix, the following observations can be made:

The number of goals scored shows the strongest correlation with total shots (0.86), big chances missed (0.77), and offsides (0.63). This suggests that the most prolific goal-scorers in the league tended to take numerous shots at goal, thereby increasing their chances of scoring. Consequently, these players also missed a significant number of opportunities. As anticipated, these high-scoring players often occupied advanced positions on the field, resulting in a greater number of offsides.

There is a relatively weak correlation between goals scored and total passes (0.18), touches (0.25), and total crosses delivered (0.28). This implies that a player's proficiency in passing and assisting had little bearing on their overall goal-scoring abilities. It seems that the most effective goal-scorers primarily focused on converting chances into goals – a testament to the enduring importance of the classic number 9 role.

To gain further insights into player performance, it is worthwhile to delve deeper into these relationships.

Further data visualization and Analysis

Let's do a scatterplot of goals vs total shots for the top 20 goalscorers.

#Scatterplot of Goals vs Total Shots fig = px.scatter(full_df[:20],x="Shots", y="Goals",title='Top 20 Goals Vs Total Shots',text="Player",template='plotly_dark',trendline="ols") fig.update_traces(textposition="top center",orientation='v',textfont_size=8.5,) fig.show()

A strong positive correlation can be observed between the number of goals scored and the total shots taken by players. For instance, Erling Haaland, who scored the most goals (35), also had the highest number of total shots (108). Similarly, Harry Kane scored 25 goals and had 120 shots. This relationship indicates that the more shots a player takes, the higher their chances of scoring goals. This could suggest that players who are more proactive and willing to take risks in front of the goal tend to score more.

Goals vs Total Shots per Goal to demonstrate the shooting efficiency of players

# Create a new column 'Shots_per_Goal' by dividing the 'Shots' column by the 'Goals' column full_df['Shots_per_Goal'] = full_df['Shots'] / full_df['Goals'] # Create a scatterplot of the top 20 players by Shots_per_Goal vs Goals fig = px.scatter(full_df[:20], x="Shots_per_Goal", y="Goals", title="Top 20 Goals Vs Total Shots per Goal", text="Player",template="plotly_dark",trendline="ols") fig.update_traces(textposition="top center", orientation="v", textfont_size=8.5) fig.show()

Calculating shots per goal (Total Shots / Goals) can provide insight into a player's shooting efficiency. A lower shots per goal ratio indicates that a player requires fewer attempts to score, demonstrating their accuracy and effectiveness in converting chances. For example, Erling Haaland's shots per goal ratio is 3.08 (108/35), while Harry Kane's is 4.8 (120/25). Comparing these ratios, Haaland is more efficient in converting his shot attempts into goals than Kane.

It is possible that Manchester City's strong midfielders are creating numerous big chances for Haaland, contributing to his increased efficiency compared to other players. To investigate this, we can plot Goals against Big Chances Missed to gain further insights.

# Create a scatterplot of the top 20 players by Shots_per_Goal vs Goals fig = px.scatter(full_df[:20], x="Big Chance Missed", y="Goals", title="Top 20 Goals Vs Big Chance Missed", text="Player", template="plotly_dark",trendline="ols") fig.update_traces(textposition="top center", orientation="v", textfont_size=8.5) fig.show()

Analyzing the relationship between goals scored and big chances missed can shed light on a player's ability to capitalize on crucial goal-scoring opportunities. A higher number of big chances missed may indicate that a player tends to squander valuable scoring chances, potentially affecting their overall goal tally. For instance, Erling Haaland scored 35 goals but missed 23 big chances, whereas Gabriel Jesus scored only 10 goals and missed 16 big chances. In this case, Haaland was able to score more goals despite missing several big chances, while Jesus' goal tally suffered due to his inability to capitalize on important scoring opportunities.

Through these 3 plots, Erling Haaland's outstanding performance across various metrics sets him apart as an outlier among his peers. His goal-scoring ability, shot-taking frequency, and efficiency in converting chances make him a highly valuable asset for his team. Although there's still room for improvement in capitalizing on big chances, Haaland's overall performance is undeniably exceptional.

Erling is exceptional in his favor - scoring but I still want to explore more about the attacking versatility which assessing a player's performance across multiple attacking metrics.

# Create a new column 'Attacking_Versatility' by adding 'Goals', 'Assists', 'Total Cross', and 'Total Through Ball' full_df['Attacking_Versatility'] = full_df['Goals'] + full_df['Assists'] + full_df['Total Cross'] + full_df['Total Through Ball'] # Sort the DataFrame by 'Attacking_Versatility' in descending order full_df = full_df.sort_values(by='Attacking_Versatility', ascending=False) # Create a scatterplot of the top 20 players by Attacking_Versatility vs Goals fig = px.scatter(full_df[:20], x="Attacking_Versatility", y="Goals", title="Top 20 Attacking Versatility Vs Goals", text="Player", template="plotly_dark", trendline="ols") fig.update_traces(textposition="top center", orientation="v", textfont_size=8.5) fig.show()

The top 20 players in terms of attacking versatility showcase a wide range of talent, including midfielders, wingers, and fullbacks, who contribute significantly to their teams' attacking prowess through crosses, through balls, and assists. Erling Haaland, despite being the league's top goal-scorer, does not feature in this top 20 list, suggesting that his contribution to his team's offense is more focused on finishing chances rather than creating them for others.

Conclusion

In this analysis, we utilized data scraped from the EPL website, cleaned and prepared it, and subsequently examined various player performance metrics. The results revealed Erling Haaland's exceptional goal-scoring abilities. His outstanding performance in scoring goals, taking shots, and efficiently converting chances positions him as a valuable asset for his team. However, Haaland's absence from the top 20 list of attacking versatility suggests that his primary contribution to the team's offense is finishing chances rather than creating opportunities for others.

The top 20 players in attacking versatility demonstrate the importance of having a diverse range of talent on a team, with midfielders, wingers, and fullbacks contributing significantly to the team's attacking success through various means, such as crosses, through balls, and assists. Overall, having a well-rounded and balanced team capable of both creating and capitalizing on goal-scoring opportunities is essential for success in the highly competitive English Premier League.

All of my codes are available on my GitHub page. You can find other projects of mine at: https://github.com/CuonggViet

Additional Plot

After completing the article, I discovered some fascinating mixed subplots using Plotly, and I recently learned about APIs. This prompted me to combine both concepts to create engaging visualizations of the Premier League data. 😄

Vocalno Database Plot.

Source: Ploty

My idea is to plot the number of goals scored by players of different nationalities to determine which nationality has the highest-scoring players. To accomplish this, we need the latitude and longitude of each country. I found Geopy provides this information through API, so let's create some code to visualize the data.

from geopy.geocoders import OpenCage api_key = '25afb28d7fe94da28398368ce7761d86' # Group the data by Nationality and calculate the total goals for each country grouped_data = full_df.groupby('Nationality')['Goals'].sum().reset_index() # Initialize a geolocator object to obtain the latitude and longitude of countries geolocator = OpenCage(api_key) # Function to get the latitude and longitude of a country def get_latitude_longitude(country): location = geolocator.geocode(country) if location is None: return pd.Series([None, None]) return pd.Series([location.latitude, location.longitude]) # Add latitude and longitude columns to the grouped_data DataFrame grouped_data[['Latitude', 'Longitude']] = grouped_data['Nationality'].apply(get_latitude_longitude) # Remove rows with missing latitude and longitude values grouped_data = grouped_data.dropna(subset=['Latitude', 'Longitude']) grouped_data

I have just created a dataset containing the number of goals scored, grouped by nationality, along with the latitude and longitude obtained from the API. Next, I'll select the top 20 records, sorted by the number of goals scored.

top_20 = grouped_data.sort_values(by='Goals', ascending=False).head(20) top_20

The data is ready. Use Ploty to make the plot.

First, organize the figures using subplots and set "rowspan" to 2 to accommodate the larger globe map. Next, plot the Scattergeo globe map to display football players' nationalities. Finally, create a bar chart to showcase the total goals scored by nationality for the top 20 countries.

import plotly.graph_objects as go from plotly.subplots import make_subplots # Initialize figure with subplots fig = make_subplots( rows=2, cols=2, column_widths=[0.6, 0.4], row_heights=[0.5, 0.5], specs=[[{"type": "scattergeo", "rowspan": 2}, {"type": "bar", "rowspan": 2}], [None, None]] ) # Add scattergeo globe map of football player nationalities fig.add_trace( go.Scattergeo(lat=grouped_data['Latitude'], lon=grouped_data['Longitude'], mode="markers", hoverinfo="text", text=grouped_data['Nationality'], showlegend=False, marker=dict(color="crimson", size=10, opacity=0.8)), row=1, col=1 ) # Add bar chart of the total goals by nationality fig.add_trace( go.Bar(x=top_20['Nationality'], y=top_20['Goals'], marker=dict(color="crimson"), showlegend=False), row=1, col=2 ) # Update geo subplot properties fig.update_geos( projection_type="orthographic", landcolor="white", oceancolor="MidnightBlue", showocean=True, lakecolor="LightBlue" ) # Set theme, margin, and annotation in layout fig.update_layout( template="plotly_dark", margin=dict(r=10, t=50, b=40, l=60), annotations=[ dict( text="Football Players' Nationalities and Total Goals", showarrow=False, xref="paper", yref="paper", x=0, y=-0.1) ] ) fig.show()