How does the employment rate in the agricultural sector impact different crop productions in different countries?

by Shan Zhou (szhou2@falcon.bentley.deu)

Welcome to the online repository of my final project for MA705. This codebase is designed to explore the research question: "How does the employment rate in the agricultural sector impact different crop productions in different countries?" It includes data preprocessing, merging techniques, and statistical analysis methods to examine the relationship between agricultural employment rates and crop yields across various nations. The datasets used are sourced from OECD. The code is structured into modules for data cleaning, merging, analysis, and visualization, ensuring ease of navigation and comprehension. This repository not only demonstrates my analytical capabilities but also serves as a resource for those interested in the intersection of labor economics and agricultural productivity.

Import Libraries and Datasets

First, import necessary libraries. We will primarily be using Pandas for data manipulation as well as Numpy, Matplotlib, Stats, and Sklearn for doing mathematical calculations, plotting figures, and modeling, respectively.

import pandas as pd import numpy as np from scipy import stats import matplotlib.pyplot as plt import seaborn as sns from sklearn.linear_model import LinearRegression from sklearn.metrics import r2_score

Next, import the necessary datasets. I used the OECD to scrape statistics for Employment by activity and Crop Production. These are the datasets I found:

df_emp: provides a detailed breakdown of employment by activity, specifically focusing on the agricultural sector. The dataset encompasses a comprehensive time frame, spanning from 1955 to 2022, and includes data from 44 countries. The employment figures are measured in thousands of individuals.

df_crop: provides an in-depth analysis of crop production across 38 countries spanning from 1990 to 2030. It specifically focuses on key staple crop: wheat, maize, rice and soybean. The data is quantified in terms of production volumes, measured in thousand tonnes for each crop.

df_emp = pd.read_csv( 'Employment by activity.csv' ) df_emp

df_crop = pd.read_csv( 'Crop production.csv' ) df_crop

Data Cleaning

# Data cleaning for 'Employment by acticity.csv' # Group up unnecessary columns columns_to_drop = ['INDICATOR', 'SUBJECT','MEASURE', 'FREQUENCY', 'Flag Codes'] # Drop unnecessary columns df_emp = df_emp.drop(columns_to_drop, axis=1) # Change number standard numeric format df_emp['Value'] = df_emp['Value'] * 1000 # Convert to interger df_emp['Value'] = df_emp['Value'].astype(int) # Change column name df_emp = df_emp.rename(columns = {'LOCATION': 'Country', 'TIME':'Year', 'Value':'Number_of_Employee'}) df_emp

#Data cleaning for 'Crop production.csv' # Group up unnecessary columns columns_to_drop = ['INDICATOR','MEASURE', 'FREQUENCY', 'Flag Codes'] # Drop unnecessary columns df_crop = df_crop.drop(columns_to_drop, axis=1) # Change number standard numeric format df_crop['Value'] = df_crop['Value'] * 1000 # Change column name df_crop = df_crop.rename(columns = {'LOCATION': 'Country', 'TIME':'Year', 'SUBJECT':'Crop_type', 'Value':'Crop_production'}) # Removing countries with all crop production values as 0 for the same crop type df_crop = df_crop[df_crop.groupby(['Country', 'Crop_type'])['Crop_production'].transform('sum') != 0] df_crop

Data Merging

Merging two datasets involves aligning them based on common attributes, in this case, country and year. The 'crop_production.csv' file encompasses data spanning from 1990 to 2030, covering 38 countries. On the other hand, 'employment_by_activity.csv' contains data from 1955 to 2022, involving 44 countries. By employing an inner merge approach, it is possible to extract and combine the overlapping data that exists in both files, thereby focusing on the countries and years common to both datasets.

# Merging the datasets on 'Country' and 'Year' df_merged = pd.merge(df_crop, df_emp, on=['Country', 'Year']) df_merged

Following the merger of the two datasets, the consolidated data now includes information from 16 countries. However, there is a noticeable variation in the starting year of crop production data for these countries. To ensure uniformity and consistency in the dataset, it is essential to select only those countries with complete data spanning from 1990 to 2022. This approach will align the data across all countries for the specified period, providing a more consistent and comparable dataset for analysis.

# Filter data for the year 1990 data_1990 = df_merged[df_merged['Year'] == 1990] # Check which countries have data for all crop types in 1990 # First, get the unique crop types unique_crops = df_merged['Crop_type'].unique() # Then, check for each country if it has all these crop types countries_with_all_crops = [] for country in data_1990['Country'].unique(): country_data = data_1990[data_1990['Country'] == country] if all(crop in country_data['Crop_type'].values for crop in unique_crops): countries_with_all_crops.append(country) df_merged = df_merged[df_merged['Country'].isin(countries_with_all_crops)] df_merged

Exploratory Data Analysis (EDA)

Time Series Analysis: This will involve examining how crop production and the number of employees in the agricultural sector have changed over time in each country.

# plotting all combinations might not be feasible within this format. def plot_time_series(df_merged): countries = df_merged['Country'].unique() crops = df_merged['Crop_type'].unique() plots = [] for country in countries: for crop in crops: # Filtering data for each country and crop country_crop_data = df_merged[(df_merged['Country'] == country) & (df_merged['Crop_type'] == crop)] fig, ax1 = plt.subplots(figsize=(10, 5)) # Plotting crop production ax1.plot(country_crop_data['Year'], country_crop_data['Crop_production'], color='tab:red', label='Crop Production') ax1.set_xlabel('Year') ax1.set_ylabel('Crop Production', color='tab:red') ax1.tick_params(axis='y', labelcolor='tab:red') # Setting up a second y-axis ax2 = ax1.twinx() ax2.plot(country_crop_data['Year'], country_crop_data['Number_of_Employee'], color='tab:blue', label='Number of Employees') ax2.set_ylabel('Number of Employees', color='tab:blue') ax2.tick_params(axis='y', labelcolor='tab:blue') plt.title(f'{crop} Production and Employees in {country}') fig.tight_layout() # Storing the plot plots.append(fig) return plots # Generating plots for all countries and crops all_plots = plot_time_series(df_merged) for plot in all_plots: plot.show()

Interpretation: 1. Inverse Relationship for Wheat: There is a notable inverse relationship between wheat production and the number of employees in the agricultural sector across various countries. As wheat production increases, the number of employees tends to decrease. 2. Trends in the United States: This inverse relationship between production and employment is consistent across different crop types in the United States, suggesting a broader trend within the country's agricultural sector. 3. Exaggerated Changes in Australia (AUS): Australian data shows significant fluctuations in both production and employment. These changes are more pronounced than in other countries. 4. Steady Decline in Japan (JPN): In Japan, both maize production and the number of employees in agriculture have been steadily decreasing over time.

To gain a deeper understanding beyond what was revealed in the exploratory data analysis (EDA), a statistical analysis was conducted to further investigate the relationships implied by the initial findings and more directly address the research question.

Correlation Analysis

When analyzing the relationship between employment rates and crop production, conducting a correlation analysis is an essential first step. This statistical method allows us to quantify the strength and direction of the association between these two variables. By calculating the Pearson correlation coefficient, we can determine whether changing in the number of employees is associated with an increase or decrease in crop production and how strong this association is. Additionally, the p-value obtained alongside the coefficient helps us understand whether the observed correlation is statistically significant. This analysis is crucial because it informs us whether a more detailed exploration, like regression analysis, is warranted. It helps to initially identify patterns or trends in the data, guiding further, more complex analyses and decision-making processes in agricultural planning and policy formulation.

# Grouping the data by both Country and Crop_type grouped_data = df_merged.groupby(['Country', 'Crop_type']) # Creating a dictionary to store the results of correlation analysis for each country-crop pair correlation_results = {} for name, group in grouped_data: # Calculating the Pearson correlation for each group corr_coeff, p_value = stats.pearsonr(group['Crop_production'], group['Number_of_Employee']) # Storing the results correlation_results[name] = {'Correlation_Coefficient': corr_coeff, 'P-value': p_value} # Converting the results dictionary to a DataFrame for better visualization df_corr = pd.DataFrame.from_dict(correlation_results, orient='index') # Add a new column 'Country-Crop' by concatenating 'Country' and 'Crop_type' df_corr['Country_Crop'] = df_corr.index.map(lambda x: '-'.join(x)) # Split the 'Country-Crop' column into 'Country' and 'Crop' df_corr[['Country', 'Crop']] = df_corr['Country_Crop'].str.split('-', expand=True) # Drop the original 'Country-Crop' column as it no longer needed df_corr = df_corr.drop(columns=['Country_Crop']) df_corr

Below is a bar graph, for better visual of the correlation result.

# Create a bar graph with crops on the x-axis, countries as separate bars, and the correlation coefficient as the bar height fig, ax = plt.subplots(figsize=(14, 8)) # Using seaborn's 'barplot' which handles the grouping automatically sns.barplot( data=df_corr, x='Crop', y='Correlation_Coefficient', hue='Country', errorbar=None ) # Customizing the plot plt.title('Correlation Coefficients for Employment Rates and Crop Production by Country and Crop Type', fontsize=16) plt.ylabel('Correlation Coefficient', fontsize=12) plt.xlabel('Type of Crop', fontsize=12) plt.xticks(rotation=45) # Rotate x-axis labels for better readability plt.legend(title='Country', bbox_to_anchor=(1.05, 1), loc='upper left') plt.grid(axis='y', linestyle='--', alpha=0.7) # Add grid lines for y-axis # Show the plot plt.tight_layout() plt.show()

Interpretation: JPN – MAIZE, KOR – RICE, and KOR-SOYBEAN shows very strong positive correlation coefficient of 0.9211, 0.8540, and 0.7667 respectively, with a p-value close to 0 indicates a very strong positive relationship between the number of employees and crop production in those countries, which is statistically significant. On other hand, KOR-WHEAT and USA-MAIZE shows very strong negative correlation coefficient of 0.7888 and 0.6733, with a p-value close to 0 indicates a very strong positive relationship between the number of employees and maize production in Korea, which is statistically significant.

Regression Analysis

Based on the result from correlation analysis, JPN - MAIZE, KOR - RICE, KOR - SOYBEAN, KOR - WHEAT, and USA - MAIZE, these pairs stand out either because of the strength of their correlations or their significance levels, indicating a more robust relationship that could be explored through regression analysis. Particularly, the pairs with very strong correlations (above 0.7 or below -0.7) are prime candidates because the employment rate is likely to be a strong predictor of crop production for these crops in these countries.

# Specified country-crop pairs for regression analysis country_crop_pairs = [('JPN', 'MAIZE'), ('KOR', 'RICE'), ('KOR', 'SOYBEAN'), ('KOR', 'WHEAT'), ('USA', 'MAIZE')] # Dictionary to store the regression results regression_results = {} # Conducting regression analysis for the specified country-crop pairs for country, crop in country_crop_pairs: # Preparing the data subset = df_merged[(df_merged['Country'] == country) & (df_merged['Crop_type'] == crop)] X = subset[['Number_of_Employee']] # Features (independent variables) y = subset['Crop_production'] # Target (dependent variable) # Performing the linear regression model = LinearRegression().fit(X, y) # Predictions and R2 score y_pred = model.predict(X) r2 = r2_score(y, y_pred) # Storing the results regression_results[f'{country} - {crop}'] = { 'Coefficient': model.coef_[0], 'Intercept': model.intercept_, 'R2 Score': r2 } # Converting the results dictionary to a DataFrame for better visualization df_reg = pd.DataFrame.from_dict(regression_results, orient='index') df_reg

Interpretation: The results of the regression analysis show different dynamics for different country-crop pairs. In Japan, there is a slight increase in maize production with additional labor, and a high R2 suggests that labor is a strong predictor of production. In Korea, labor has a substantial positive impact on rice production, with a significant R2 indicating its importance. However, for soybean production in Korea, the impact of labor is less pronounced and other factors are also important. Surprisingly, increased labor seems to have a negative effect on wheat production in Korea, suggesting possible inefficiencies. In the US, corn production has a strongly negative coefficient, suggesting a decrease in production per additional worker, possibly due to overstaffing or technological progress. The relatively low R2 for the US suggests that factors other than labor have a significant impact on production levels. These results underscore the complexity of the relationship between labor and agricultural production, which varies considerably across crops and countries.