Day 14

import pandas as pd import numpy as np import nltk

tweets = pd.read_csv('/work/tweets_labelled_09042020_16072020.csv', sep=';')

tweets.head() nltk.download('vader_lexicon')

[nltk_data] Downloading package vader_lexicon to /root/nltk_data...

from nltk.sentiment.vader import SentimentIntensityAnalyzer

sid = SentimentIntensityAnalyzer()

sentence = "It was the best of times."

ss = sid.polarity_scores(sentence)

for k in sorted(ss): print('{0}: {1}, '.format(k, ss[k]), end='') print()

compound: 0.6369, 
neg: 0.0, 
neu: 0.543, 
pos: 0.457,

df = pd.DataFrame({'text': ["This is great!", "This is bad.", "This is neither.", ]})

df

textobject

0

This is great!

1

This is bad.

2

This is neither.

df['sent'] = df.text.apply(sid.polarity_scores) for i in ['compound','neg','neu','pos']: df[i] = df['sent'].apply(lambda x: x[i]) df.drop('sent',axis=1)

textobject

compoundfloat64

0

This is great!

0.6588

1

This is bad.

-0.5423

2

This is neither.

0

tweets = pd.read_csv('data/stocktweets/tweets_labelled_09042020_16072020.csv', sep=';')

There’s an error in this block

Try running the app again, or contact the app’s creator

tweets.head()

There has been an error in the blocks above

Run the app again or contact app’s creator

tweets['sent'] = tweets.text.apply(sid.polarity_scores) for i in ['compound','neg','neu','pos']: tweets[i] = tweets['sent'].apply(lambda x: x[i]) tweets = tweets.drop('sent',axis=1)

There has been an error in the blocks above

Run the app again or contact app’s creator

tweets['vader'] = np.where(tweets.compound < -0.1, 'negative', np.where(tweets.compound > 0.1, 'positive', 'neutral')) tweets

There has been an error in the blocks above

Run the app again or contact app’s creator

t = tweets.dropna() len(t[t['sentiment'] == t['vader']])/len(t)

There has been an error in the blocks above

Run the app again or contact app’s creator

pos_or_neg = tweets.query("sentiment == 'positive' | sentiment == 'negative'").copy()

There has been an error in the blocks above

Run the app again or contact app’s creator

from nltk.tokenize import TweetTokenizer tt = TweetTokenizer() df['text'].apply(tt.tokenize)

There has been an error in the blocks above

Run the app again or contact app’s creator

pos_or_neg['words'] = pos_or_neg['text'].str.lower().apply(tt.tokenize)

There has been an error in the blocks above

Run the app again or contact app’s creator

pos_or_neg.head()

There has been an error in the blocks above

Run the app again or contact app’s creator

from nltk.stem import WordNetLemmatizer

There has been an error in the blocks above

Run the app again or contact app’s creator

lemmatizer = WordNetLemmatizer()

There has been an error in the blocks above

Run the app again or contact app’s creator

pos_or_neg['lemmas'] = pos_or_neg['words'].apply(lambda words: [lemmatizer.lemmatize(word) for word in words])

There’s an error in this block

Try running the app again, or contact the app’s creator

# nltk.download('omw-1.4')

There has been an error in the blocks above

Run the app again or contact app’s creator

positive = pos_or_neg.query('sentiment == "positive"') negative = pos_or_neg.query('sentiment == "negative"')

There has been an error in the blocks above

Run the app again or contact app’s creator

pos_words = set(positive.explode('lemmas')['lemmas']) neg_words = set(negative.explode('lemmas')['lemmas'])

There’s an error in this block

Try running the app again, or contact the app’s creator

len(pos_words)

There has been an error in the blocks above

Run the app again or contact app’s creator

len(neg_words)

There has been an error in the blocks above

Run the app again or contact app’s creator

pos_only = pos_words.difference(neg_words) neg_only = neg_words.difference(pos_words)

There has been an error in the blocks above

Run the app again or contact app’s creator