Welcome to my Notebook!

I love the transformers library. It is by far the easiest way to get started using transformers for NLP, which is the currently the bleeding edge

The first step is grabbing the model and the tokenizer from the transformers library

import tensorflow as tf import tensorflow_hub as hub from transformers.modeling_tf_openai import TFOpenAIGPTLMHeadModel #this is the GPT transformer with additional layers added for easy language modeling from transformers.tokenization_openai import OpenAIGPTTokenizer import simpletransformers

Don't worry if the following cell takes some time! The model just needs a minute to download.

model = TFOpenAIGPTLMHeadModel.from_pretrained("openai-gpt") tokenizer = OpenAIGPTTokenizer.from_pretrained("openai-gpt")

Note: GPT2 is a newer transformer model for text generation, but the library only has support for GPT currently for text generation.

prompt_text = "I could not think of a clever prompt because"

^^ This is the fun part! Go wild! Pick the prompt text!

encoded_prompt = tokenizer.encode(prompt_text,add_special_tokens=False,return_tensors='tf') encoded_prompt

Since transformers supplies a tokenizer for the GPT model, this was a much easier solution than using the universal sentence encoder. The tensors return as type pytorch as opposed to tensorflow because I find that I run into fewer bugs when that is the case (this might be a result that the library was originally written for pytorch).

num_sequences = 1 # this is the number of different sequences/ sentences the generator will create given the prompt length = 15 # if you want to strictly follow the prompt for this question set the length to one. If you want to see how creative GPT can be feel free to amend

Let's get some sequences!

generated_sequences = model.generate( input_ids=encoded_prompt, do_sample=True, max_length=length + len(encoded_prompt[0]), temperature=1.0, top_k=5, top_p=0.9, repetition_penalty=1.0, )

There is a lot to unpack in that cell. Credit where credit is due- the library has fantastic documentation, so lets unpack.

Input ids is simply specifying what the input will be.

Do_sample - this prevents the model from just picking the most likely word at every step greedily.

Max_len - what is the length of sequence we want?

Temperature - I find this one really interesting. This is a measure of how risky our model will be when picking words. Feel free to tweak this!

Top_k and top_p are similar in that they limit the number of words the model considers when decoding before randomly sampling from the word probabilities.

The repetition penalty is meant to avoid sentences that repeat themselves without anything really interesting.

Kudos to both the docs https://huggingface.co/transformers/main_classes/model.html?highlight=tfpretrained#transformers.TFPreTrainedModel

and this excellent medium post for helping me understand the nitty gritty here.

https://medium.com/voice-tech-podcast/visualising-beam-search-and-other-decoding-algorithms-for-natural-language-generation-fbba7cba2c5b

for sequence in generated_sequences: text = tokenizer.decode(sequence, clean_up_tokenization_spaces=True) print(text)

And tada! We have predicted text. However, that is not the only way to generate text. By far the simplest way is with a pipeline.

from transformers import pipeline generator = pipeline("text-generation") text_1 = generator("Text generation is cool because", max_length=50) text_1[0]['generated_text']

Another great tool in language modeling in the Universal sentence encoder. It will take any sentence and return a size 512 vector encoding it. Then you can perform operations like cosine similarity to see how similar they are.

embed = hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

embeddings = embed([ "the circus is filled with a variety of exotic animals", "the ringleader had the lion jump through the hoop", "excel is a popular tool in many business environments"]) embeddings

And out pop our embeddings. Let's check if our first 2 sentences are more closely related than the last 2.

from scipy import spatial result_1 = 1 - spatial.distance.cosine(embeddings[0], embeddings[1]) result_2 = 1 - spatial.distance.cosine(embeddings[1], embeddings[2]) result_3 = 1 - spatial.distance.cosine(embeddings[0], embeddings[2]) print(result_1,result_2,result_3)

The first 2 sentences are definitely more closely related than any other combination by a factor of 2 and 20 respectively.

def UniversalEmbedding(x): return embed(tf.squeeze(tf.cast(x, tf.string)), signature="default", as_dict=True)["default"]

def make_model(vocab_size): model = Sequential() model.add(layers.Lambda(UniversalEmbedding,output_shape=(embed_size,))) model.add(LSTM(100, return_sequences=True)) model.add(LSTM(100)) model.add(Dropout(0.2)) model.add(Dense(100, activation='relu')) model.add(Dense(vocab_size, activation='softmax')) return model

Given a dataset to train on and then some text prompt, this model would use the Universal Sentence Embedding to generate new text. However, pretrained models are more efficient to use and the results will be much stronger so the transformers library is preferable to use.

References and Credit:

https://github.com/ageron/handson-ml2 - Great textbook with tons of useful code samples for near any machine learning project

https://huggingface.co/transformers/ - Previously mentioned but still a great resource for everything NLP

Conclusion

Thanks for reading all of my code this way through. If you have any questions regarding the code included within feel free to reach out to me at josh.zwiebel@uwaterloo.ca

Josh Zwiebel

Welcome to my Notebook!

^^ This is the fun part! Go wild! Pick the prompt text!

Conclusion

Recommended on Deepnote

Stock Market Analysis

The 10 Best Ways to Create NumPy Arrays

Wide Residual Networks