The Witcher Network Map

Introduction

The world of "The Witcher", crafted by Andrzej Sapkowski, is an epic fantasy series that has made waves in the realms of literature, gaming, and entertainment alike. This series, with its rich narrative and complex characters, has been the foundation for several adaptations, including video games and a Netflix original series. In particular, the latest season, "The Witcher 2," has sparked a renewed interest and enthusiasm in audiences worldwide. After immersing myself in the second season of this gripping series, I was compelled to delve deeper into its intricacies, leading to the inception of this research project.

Objectives:

This research aims to provide a comprehensive exploration of "The Witcher 2," focusing primarily on the series' key characters and communities. By analyzing these elements, I seek to enhance our understanding of the series' narrative structure and its portrayal of a multifaceted universe. The objectives of this research are:

Who are the most important characters in "The Witcher 2" series?

What are the main communities depicted in the series?

Planning:

Use Selenium, a web scraping tool, to extract character names from The Witcher Wiki.

Process the data by converting the books into sentences and using Named Entity Recognition (NER) to identify and classify characters' named entities.

Define a "box" of sentences in which the characters' names are mentioned between these sentences. If this condition is met, we can infer that the characters mentioned in these sentences have a relationship with each other.

Use Pyvis, a Python library for visualizing networks, to plot the relationships between characters and communities. This will allow us to gain a better understanding of the complex web of relationships that exists within The Witcher series.

Extract character names

This Witcher fandom had created all the Characters in the story. We will use Selenium to scrap all character names and the books.

# Configure Chrome options chrome_options = Options() chrome_options.add_argument("--headless") chrome_options.add_argument("--no-sandbox") # Disable logging from webdriver_manager logging.getLogger('WDM').setLevel(logging.NOTSET) os.environ['WDM_LOG'] = 'False' # Set up webdriver service and driver webdriver_service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome(service=webdriver_service, options=chrome_options) # Set the page URL and navigate to it page_url = "https://witcher.fandom.com/wiki/Category:Characters_in_the_stories" driver.get(page_url) # Find all book categories on the page and extract their URLs and names book_categories = driver.find_elements(by=By.CLASS_NAME, value='category-page__member-link') books = [] for category in book_categories: book_url = category.get_attribute('href') book_name = category.text books.append({'book_name': book_name, 'book_url': book_url})

# Define an empty list to store characters characters = [] # Iterate over each book and extract character information for book in books: # Navigate to the book page driver.get(book['book_url']) # Find all character elements on the page and extract their names character_elems = driver.find_elements(by=By.CLASS_NAME, value='category-page__member-link') # Iterate over each character element and add its name to the list of characters for elem in character_elems: characters.append({'book': book['book_name'], 'character': elem.text})

df = pd.DataFrame(characters) df

Plot a chart of Character count for each book

# Convert the list of characters to a pandas dataframe character_df = pd.DataFrame(characters) # Plot a bar chart of the character count for each book character_df['book'].value_counts().plot(kind='bar') # Display the plot plt.show()

We have located a valuable resource on GitHub, which includes text files of all books in the Witcher series. Our next step will involve importing these files for further processing. We plan to utilize Named Entity Recognition (NER) for identifying and classifying the named entities of all characters present within these texts.

# Load spacy English language model import spacy NER = spacy.load("en_core_web_sm") import os # Get all book files in the data directory all_books = [b for b in os.scandir('data') if '.txt' in b.name] all_books

The Witcher 2 series on Netflix is based on the plot of The Blood of Elves Novel. So we will put this book to examine the character's named entities.

# Select the book file in the list and read its contents book = [b for b in all_books if 'I - Blood of Elves.txt' in b.name][0] with open(book, 'r') as f: book_text = f.read() # Pass the book text to the spaCy model to extract named entities book_doc = NER(book_text)

Now the rendered visualization will show the identified named entities in the text as highlighted phrases, with a label indicating the type of entity (e.g., person, organization, location, etc.)

# Display the identified entities using the displacy module displacy.render(book_doc[700:1700], style="ent", jupyter=True)

Clean the data. Remove some brackets and takes the first name of the characters

# Read characters character_df = pd.read_csv("df.csv") # Remove brackets and text within brackets import re character_df['character'] = character_df['character'].apply(lambda x: re.sub("[\(].*?[\)]", "", x)) character_df['character_firstname'] = character_df['character'].apply(lambda x: x.split(' ', 1)[0]) character_df

Get named entity list per sentence

Next, I will go sentence by sentence through the books and create a list of entities for those sentences

# Initialize an empty list to store sentence entities sent_entity_df = [] # Loop through sentences in the document and store named entity list for each sentence for sent in book_doc.sents: entity_list = [ent.text for ent in sent.ents] sent_entity_df.append({"sentence": sent, "entities": entity_list}) # Create a pandas DataFrame from the list of sentence entities sent_entity_df = pd.DataFrame(sent_entity_df) # Display the DataFrame sent_entity_df

# Define a function to filter out non-character entities def filter_entity(ent_list, character_df): return [ent for ent in ent_list if ent in list(character_df.character) or ent in list(character_df.character_firstname)]

# Apply the filter_entity function to the entities in each sentence and create a new column sent_entity_df['character_entities'] = sent_entity_df['entities'].apply(lambda x: filter_entity(x, character_df)) # Filter out sentences that don't have any character entities sent_entity_df_filtered = sent_entity_df[sent_entity_df['character_entities'].map(len) > 0] # Display the filtered DataFrame sent_entity_df_filtered

unique_entities = list(set([entity for sublist in sent_entity_df_filtered["character_entities"] for entity in sublist])) unique_entities

We see that ['Geralt of Rivia'] and ['Geralt'] identify as 2 different entities. It will cause a false relationship between entities. Our solution only takes the first character just to be consistent.

sent_entity_df_filtered["character_entities"]= sent_entity_df_filtered["character_entities"].apply(lambda x: [item.split()[0] for item in x]) sent_entity_df_filtered

Create relationships

Now it is an important part. I will extract the character entities from each sentence in a window of 5 sentences, remove duplicates, and create relationships between adjacent characters. The relationships are stored in a DataFrame named relationship_df, which is displayed at the end of the code.

# Define window size and initialize empty list for relationships window_size = 5 relationships = [] # Loop through each index in the filtered DataFrame for i in range(sent_entity_df_filtered.index[-1]): # Set the end index of the window end_i = min(i+5, sent_entity_df_filtered.index[-1]) # Extract the character entities from the sentences within the window char_list = sum((sent_entity_df_filtered.loc[i: end_i].character_entities), []) # Remove duplicated characters that are next to each other char_unique = [char_list[i] for i in range(len(char_list)) if (i==0) or char_list[i] != char_list[i-1]] # Create relationships between each pair of adjacent characters if len(char_unique) > 1: for idx, a in enumerate(char_unique[:-1]): b = char_unique[idx + 1] relationships.append({"source": a, "target": b}) # Create DataFrame from the relationships list relationship_df = pd.DataFrame(relationships) relationship_df

I sorted the character alphabetically so that the relationship between Ciri to Geralt and Geralt to Ciri is the same. Then count the relationships between them.

# Sort the cases with a->b and b->a relationship_df = pd.DataFrame(np.sort(relationship_df.values, axis = 1), columns = relationship_df.columns) relationship_df

relationship_df["value"] = 1 relationship_df = relationship_df.groupby(["source","target"], sort=False, as_index=False).sum() relationship_df

Graph analysis and visualization

# define G G = nx.from_pandas_edgelist(relationship_df, source = "source", target = "target", edge_attr = "value", create_using = nx.Graph()) plt.figure(figsize=(15,12)) pos = nx.kamada_kawai_layout(G) nx.draw(G, with_labels=True, node_color='skyblue', edge_cmap=plt.cm.Blues, pos = pos) plt.show()

Network visualization

from pyvis.network import Network net = Network(notebook = True, bgcolor='#222222', font_color='white',cdn_resources='remote', width="1400px", height="1000px") # Define node size node_degree = dict(G.degree) #Setting up node size attribute nx.set_node_attributes(G, node_degree, 'size') net.from_nx(G) net.show("witcher.html") display_html('witcher.html')

degree_dict = nx.degree_centrality(G) degree_dict

degree_df = pd.DataFrame.from_dict(degree_dict, orient='index', columns=['centrality']) # Plot top 10 nodes degree_df.sort_values('centrality', ascending=False)[0:9].plot(kind="bar")

# Betweenness centrality betweenness_dict = nx.betweenness_centrality(G) betweenness_df = pd.DataFrame.from_dict(betweenness_dict, orient='index', columns=['centrality']) # Plot top 10 nodes betweenness_df.sort_values('centrality', ascending=False)[0:9].plot(kind="bar")

# Closeness centrality closeness_dict = nx.closeness_centrality(G) closeness_df = pd.DataFrame.from_dict(closeness_dict, orient='index', columns=['centrality']) # Plot top 10 nodes closeness_df.sort_values('centrality', ascending=False)[0:9].plot(kind="bar")

# Save centrality measures nx.set_node_attributes(G, degree_dict, 'degree_centrality') nx.set_node_attributes(G, betweenness_dict, 'betweenness_centrality') nx.set_node_attributes(G, closeness_dict, 'closeness_centrality') #Community detection import community as community_louvain communities = community_louvain.best_partition(G) communities nx.set_node_attributes(G, communities, 'group') com_net = Network(notebook = True, width="1300px", height="1000px", bgcolor='#222222', font_color='white') com_net.from_nx(G) com_net.show("witcher_communities.html") display_html('witcher_communities.html')

Conclusion

In conclusion, throughout this research article, we have delved into the complex and captivating world of The Witcher 2 series to identify and analyze the most significant characters and communities that exist within its realm. Utilizing a combination of qualitative analysis and data visualization through Pyvis, we have determined that the most important characters in the series are Geralt, Ciri, and Yennefer.

These characters not only drive the narrative but also play critical roles in shaping the different communities in the universe. By plotting the relationships between characters, we have been able to gain a deeper understanding of the intricate connections and alliances that form the backbone of this richly detailed world. Furthermore, this analysis has provided valuable insights into the dynamics and intricacies of the various communities that populate The Witcher 2 series, highlighting the importance of these key characters in the broader narrative context.