Twitter Analysis - Cleaning

Libraries Used

Installing third party libraries

!pip install openpyxl !pip install tweet-preprocessor

Requirement already satisfied: openpyxl in /root/venv/lib/python3.9/site-packages (3.1.2)
Requirement already satisfied: et-xmlfile in /root/venv/lib/python3.9/site-packages (from openpyxl) (1.1.0)
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/root/venv/bin/python -m pip install --upgrade pip' command.
Collecting tweet-preprocessor
  Downloading tweet_preprocessor-0.6.0-py3-none-any.whl (27 kB)
Installing collected packages: tweet-preprocessor
Successfully installed tweet-preprocessor-0.6.0
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/root/venv/bin/python -m pip install --upgrade pip' command.

Specific corpuses downloaded from NLTK

import nltk nltk.download('wordnet') nltk.download('omw-1.4')

[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data] Downloading package omw-1.4 to /root/nltk_data...

import pandas as pd import preprocessor as p from nltk.tokenize import TweetTokenizer from nltk.stem import WordNetLemmatizer import re import string from scipy.special import softmax import numpy as np

Importing Tweet File

Importing Tweets

Tweets were extracted using RapidMiner. Extracted tweets were exported as Excel file.

airasia_twts = pd.read_excel("airasia3.xlsx")

/root/venv/lib/python3.9/site-packages/openpyxl/styles/stylesheet.py:226: UserWarning: Workbook contains no default style, apply openpyxl's default
  warn("Workbook contains no default style, apply openpyxl's default")

airasia_twts.columns

Removing unwanted columns

Keeping only the text and retweets columns, the rest were removed

airasia_twts.drop(columns=["Created-At", "From-User", "From-User-Id", "To-User","To-User-Id", "Language", "Source", "Geo-Location-Latitude", "Geo-Location-Longitude", "Id" ], inplace=True)

Text Pre-processing

Initialize tokenizers

NLTK has a dedicated tokenizer for Tweets which was used in this project. Inflection of words in Tweets were lemmatized using Lemmatizer

tknzr = TweetTokenizer(preserve_case=False) lemmatizer = WordNetLemmatizer()

Custom function for cleaning text

Cleaning common tokens in a Tweet such as RT, @ using a library called tweet-preprocessor.

Removing punctuations and digits

Lemmatizing words

def preprocess_tweets(tweet: str): cleaned_text = p.clean(tweet) translation_table = str.maketrans('', '', string.punctuation+string.digits) cleaned_text = cleaned_text.translate(translation_table) cleaned_text = [(lemmatizer.lemmatize(w)) for w in tknzr.tokenize((cleaned_text))] return ' '.join(cleaned_text)

airasia_twts.head()

Textobject

Retweet-Countint64

SAF produced in India helps power AirAsia flight https://t.co/fUWPznKdyK

Hello @airasia refund tic yg beli dkt ur website from jogja to jakarta still not responding...then simply said case close..d flight @Citilink hell not available for month.. @airasia @airasia @airasia #senangbuatduit #viral #airasia #malaysiaviral

@airasia @foodpanda_my My 2 overseas flights in 2020 cancelled by AirAsia due to Covid. Already 3 years now and still no cash refunds despite numerous emails to AAX n MAVCOM with passengers forced to accept only useless travel vouchers without any choice or option. PAY OUR FULL CASH REFUNDS NOW!

@airasia Hi, i need to cancel a ticket and get a refund but i can't do it on you app or web page either, do you have a real person as a customer service instead of a chatbot please? An email address would also help. My flight will depart from Hong Kong. Thank You.

@GyeoulHp @TPolah05 Okay lang ang air asia for me!! (chsr singit) actually naka try ako ng airasia just this month bc my cebpac flight was cancelled for no apparent reason tapos i have to fly to mnl on that day dapat. Maganda rin siguro experience ko kasi first flight ako hehe

Applying the cleaning function to Tweet column

airasia_twts['Text'] = airasia_twts['Text'].apply(lambda x: preprocess_tweets(x))

Removing duplicated Tweets

Mostly commonly the tweet that has been retweeted were removed

airasia_twts.drop_duplicates(subset=["Text"], inplace=True)

Exporting cleaned file

airasia_twts.to_csv("cleaned_airasia_tweets_1.csv")

.css-ytumd6{-webkit-text-decoration:none;text-decoration:none;}