Pierian Training
Project Management Academy
Six Sigma Online
United Training
Velopi
Watermark Learning
Login
Welcome to this blog post on Natural Language Processing, specifically generating new text! In this blog post we’ll explore how to gather a bunch of Weezer lyrics and train a Recurrent Neural Network (RNN) to generate new lyrics character by character.
Let’s dive in!
Gathering the Data¶
Let’s leverage an existing an API to gather the lyrics, www.genius.com provides an API we can use, we just need to register an account and then visit: https://genius.com/api-clients and authorize a new API client. (Note: You need to provide a placeholder URL that includes https://wwww).
You will then be provided with a CLIENT ID and a secret key which will look something like:
rux5zYOIGi-9kHDSU8IYhbbbGaxTgopj-uR10K2TKPcYhbbbGaxT
Also click the button on the page to generate an access token.
Now we use the power of the Python ecosystem to discover that people have already created Python interfaces to easily connect with these API keys.
https://lyricsgenius.readthedocs.io/en/master/
We just need to install the library:
pip install lyricsgenius!pip install lyricsgenius
from lyricsgenius import Genius
token = input("Paste your token here: ")
Let’s use the Python library and API to grab all of Weezer’s lyrics. We should note, depending on how many songs you query you may get data contamination due to Live recordings duplicating data, Rivers Cuomo’s solo work being labeled as Weezer, and Weezer’s Teal Album) which is actually a bunch of 80s covers. We could clean this up, but for now we’ll just treat it all as Weezer lyrics. We’ll limit this to 200 songs.
genius = Genius(token)
artist = genius.search_artist('Weezer',max_songs=200)
Let’s quickly save these lyrics to a JSON file.
artist.save_lyrics('Weezer200.json')
Wrote Weezer200.json.
We can then loop through all the songs and grab the lyrics, let’s explore the first few and explore the structure:
for song in artist.songs[:2]:
print(song.lyrics)
Say It Ain’t So Lyrics[Intro] Oh, yeah Alright [Verse 1] Somebody's Heine Is crowding my icebox Somebody's cold one Is giving me chills Guess I'll just close my eyes [Pre-Chorus] Oh, yeah Alright Feels good Inside [Verse 2] Flip on the telly Wrestle with Jimmy Somethin' is bubblin' Behind my back The bottle is ready to blow [Chorus] Say it ain't so Your drug is a heartbreaker Say it ain't so My love is a life taker [Verse 3] I can't confront you I never could do That which might hurt you So try and be cool When I say "This way Is a water slide away from me that takes you further every day" So be cool [Chorus] Say it ain't so Your drug is a heartbreaker Say it ain't so My love is a life taker [Bridge] Dear daddy, I write you In spite of years of silence You've cleaned up, found Jesus Things are good, or so I hear This bottle of Stephen's Awakens ancient feelin's Like father, stepfather The son is drowning in the flood Yeah, yeah-yeah, yeah-yeah [Guitar Solo] [Chorus] Say it ain't so Your drug is a heartbreaker Say it ain't so My love is a life taker77Embed Buddy Holly Lyrics[Verse 1] What's with these homies dissin' my girl? Why do they gotta front? What did we ever do to these guys That made them so violent? [Pre-Chorus] Woo-hoo, but you know I'm yours Woo-hoo, and I know you're mine Woo-hoo, and that's for all of time [Chorus] Ooh wee ooh, I look just like Buddy Holly Oh oh, and you're Mary Tyler Moore I don't care what they say about us anyway I don't care 'bout that [Verse 2] Don't you ever fear, I'm always near I know that you need help Your tongue is twisted, your eyes are slit You need a guardian [Pre-Chorus] Woo-hoo, and you know I'm yours Woo-hoo, and I know you're mine Woo-hoo, and that's for all of time [Chorus] Ooh wee ooh, I look just like Buddy Holly Oh oh, and you're Mary Tyler Moore I don't care what they say about us anyway I don't care 'bout that I don't care 'bout that [Bridge] Bang, bang, knocking on the door Another big bang, get down on the floor Oh no, what do we do? Don't look now, but I lost my shoe I can't run and I can't kick What's a matter, babe, are you feelin' sick? What's a matter, what's a matter, what's a matter you? What's a matter, babe, are you feelin' blue? Oh-oh [Guitar Solo] [Pre-Chorus] And that's for all of time And that's for all of time [Chorus] Ooh wee ooh, I look just like Buddy Holly Oh oh, and you're Mary Tyler Moore I don't care what they say about us anyway I don't care 'bout that I don't care 'bout that I don't care 'bout that I don't care 'bout that39Embed
Notice the use of whitespace and terms such as [Chorus] and [Bridge] to define the song structure.
We now have two options for tokenization, we can tokenize words, for example:
lyrics = 'I look just like Buddy Holly'
tokens = lyrics.split()
tokens
['I', 'look', 'just', 'like', 'Buddy', 'Holly']
Or we could tokenize by character:
lyrics = "I look just like Buddy Holly"
list(lyrics)
['I', ' ', 'l', 'o', 'o', 'k', ' ', 'j', 'u', 's', 't', ' ', 'l', 'i', 'k', 'e', ' ', 'B', 'u', 'd', 'd', 'y', ' ', 'H', 'o', 'l', 'l', 'y']
You’ll notice whitespace and characters such as ‘\n’ denoting a new line will be kept if we tokenize by individual characters. While you should expect more human like results if we tokenize by words, let’s push the limits of TensorFlow and RNNs by exploring just how well a simple Recurrent Neural Network can work if we try to have it predict characters. You should note that for the purposes of RNN text generation this is a tiny dataset. Let’s go through some text processing to create a numeric representation of the letters through text vectorization.
Text Processing¶
We know a neural network can’t take in the raw string data, we need to assign numbers to each character. Let’s create two dictionaries that can go from numeric index to character and character to numeric index.
import numpy as np
text = ''
# This will remove special characters for songs that are not in english
special_chars = set([ 'ँ', 'आ', 'ए', 'क', 'ग', 'ज', 'त',
'द', 'ध', 'न', 'प', 'ब', 'म', 'य', 'र', 'व', 'श', 'ह', 'ा', 'ि',
'ी', 'ू', 'े', 'ै', 'ो', '्', '\u2005', '\u200c', '—', '‘', '’',
'\u205f', '느', '사', '어', '이', '제', '죠', '품', '회','\u0435','\xa0'])
for song in artist.songs:
chars =set(song.lyrics)
if len(chars.intersection(special_chars)) == 0:
text += song.lyrics
else:
pass
f = open("200_Weezer_Song_Lyrics.txt","w")
f.write(text)
f.close()
Now you can always read in the txt data with:
# Read in cleaned text
with open('200_Weezer_Song_Lyrics.txt') as f:
text = f.read()
# The unique characters in the file
vocab = sorted(set(text))
char_to_ind = {u:i for i, u in enumerate(vocab)}
# char_to_ind
ind_to_char = np.array(vocab)
ind_to_char
array(['\n', ' ', '!', '"', '&', "'", '(', ')', '*', ',', '-', '.', '/',
'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '?',
'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'Y', 'Z', '[',
']', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l',
'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y',
'z', '¡', 'ó'], dtype='<U1')
encoded_text = np.array([char_to_ind[c] for c in text])
encoded_text
array([27, 73, 56, ..., 54, 57, 56])
We now have a mapping we can use to go back and forth from characters to numerics.
sample = text[:20]
sample
'Buddy Holly Lyrics[V'
encoded_text[:20]
array([27, 73, 56, 56, 77, 1, 33, 67, 64, 64, 77, 1, 37, 77, 70, 61, 55,
71, 51, 47])
Creating Batches¶
Overall what we are trying to achieve is to have the model predict the next highest probability character given a historical sequence of characters. Its up to us (the user) to choose how long that historic sequence. Too short a sequence and we don’t have enough information (e.g. given the letter “a” , what is the next character) , too long a sequence and training will take too long and most likely overfit to sequence characters that are irrelevant to characters farther out. While there is no correct sequence length choice, you should consider the text itself, how long normal phrases are in it, and a reasonable idea of what characters/words are relevant to each other.
print(text[:500])
Buddy Holly Lyrics[Verse 1] What's with these homies dissin' my girl? Why do they gotta front? What did we ever do to these guys That made them so violent? [Pre-Chorus] Woo-hoo, but you know I'm yours Woo-hoo, and I know you're mine Woo-hoo, and that's for all of time [Chorus] Ooh wee ooh, I look just like Buddy Holly Oh oh, and you're Mary Tyler Moore I don't care what they say about us anyway I don't care 'bout that [Verse 2] Don't you ever fear, I'm always near I know that you need help You
line = "Say it ain't so"
len(line)
15
Bringing in TensorFlow¶
Now its time to start shaping our data to work with TensorFlow
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Training Sequences¶
The actual text data will be the text sequence shifted one character forward. For example:
Sequence In: “Hello my nam” Sequence Out: “ello my name”
We can use the tf.data.Dataset.from_tensor_slices function to convert a text vector into a stream of character indices.
seq_len = 120
total_num_seq = len(text)//(seq_len+1)
total_num_seq
1578
# Create Training Sequences
char_dataset = tf.data.Dataset.from_tensor_slices(encoded_text)
# for i in char_dataset.take(500):
# print(ind_to_char[i.numpy()])
The batch method converts these individual character calls into sequences we can feed in as a batch. We use seq_len+1 because of zero indexing. Here is what drop_remainder means:
drop_remainder: (Optional.) A tf.bool scalar tf.Tensor, representing
whether the last batch should be dropped in the case it has fewer than
batch_size elements; the default behavior is not to drop the smaller
batch.
sequences = char_dataset.batch(seq_len+1, drop_remainder=True)
Now that we have our sequences, we will perform the following steps for each one to create our target text sequences:
- Grab the input text sequence
- Assign the target text sequence as the input text sequence shifted by one step forward
- Group them together as a tuple
def create_seq_targets(seq):
input_txt = seq[:-1]
target_txt = seq[1:]
return input_txt, target_txt
dataset = sequences.map(create_seq_targets)
for input_txt, target_txt in dataset.take(1):
print(input_txt.numpy())
print(''.join(ind_to_char[input_txt.numpy()]))
print('\n')
print(target_txt.numpy())
# There is an extra whitespace!
print(''.join(ind_to_char[target_txt.numpy()]))
[27 73 56 56 77 1 33 67 64 64 77 1 37 77 70 61 55 71 51 47 57 70 71 57 1 14 52 0 48 60 53 72 5 71 1 75 61 72 60 1 72 60 57 71 57 1 60 67 65 61 57 71 1 56 61 71 71 61 66 5 1 65 77 1 59 61 70 64 25 0 48 60 77 1 56 67 1 72 60 57 77 1 59 67 72 72 53 1 58 70 67 66 72 25 0 48 60 53 72 1 56 61 56 1 75 57 1 57 74 57 70 1 56 67 1 72 67 1 72 60] Buddy Holly Lyrics[Verse 1] What's with these homies dissin' my girl? Why do they gotta front? What did we ever do to th [73 56 56 77 1 33 67 64 64 77 1 37 77 70 61 55 71 51 47 57 70 71 57 1 14 52 0 48 60 53 72 5 71 1 75 61 72 60 1 72 60 57 71 57 1 60 67 65 61 57 71 1 56 61 71 71 61 66 5 1 65 77 1 59 61 70 64 25 0 48 60 77 1 56 67 1 72 60 57 77 1 59 67 72 72 53 1 58 70 67 66 72 25 0 48 60 53 72 1 56 61 56 1 75 57 1 57 74 57 70 1 56 67 1 72 67 1 72 60 57] uddy Holly Lyrics[Verse 1] What's with these homies dissin' my girl? Why do they gotta front? What did we ever do to the
Generating training batches¶
Now that we have the actual sequences, we will create the batches, we want to shuffle these sequences into a random order, so the model doesn’t overfit to any section of the text, but can instead generate characters given any seed text.
# Batch size
batch_size = 128
# Buffer size to shuffle the dataset so it doesn't attempt to shuffle
# the entire sequence in memory. Instead, it maintains a buffer in which it shuffles elements
buffer_size = 10000
dataset = dataset.shuffle(buffer_size).batch(batch_size, drop_remainder=True)
dataset
<BatchDataset element_spec=(TensorSpec(shape=(128, 120), dtype=tf.int32, name=None), TensorSpec(shape=(128, 120), dtype=tf.int32, name=None))>
Creating the Model¶
We will use an LSTM based model with a few extra features, including an embedding layer to start off with and two LSTM layers. We based this model architecture off the DeepMoji and the original source code can be found here.
The embedding layer will serve as the input layer, which essentially creates a lookup table that maps the numbers indices of each character to a vector with “embedding dim” number of dimensions. As you can imagine, the larger this embedding size, the more complex the training. This is similar to the idea behind word2vec, where words are mapped to some n-dimensional space. Embedding before feeding straight into the LSTM usually leads to more realisitic results.
# Length of the vocabulary in chars
vocab_size = len(vocab)
# The embedding dimension
embed_dim = 64
# Number of RNN units
rnn_neurons = 1026
Now let’s create a function that easily adapts to different variables as shown above.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM,Dense,Embedding,Dropout,GRU
Setting up Loss Function¶
For our loss we will use sparse categorical crossentropy, which we can import from Keras. We will also set this as logits=True
from tensorflow.keras.losses import sparse_categorical_crossentropy
help(sparse_categorical_crossentropy)
Help on function sparse_categorical_crossentropy in module keras.losses:
sparse_categorical_crossentropy(y_true, y_pred, from_logits=False, axis=-1)
Computes the sparse categorical crossentropy loss.
Standalone usage:
>>> y_true = [1, 2]
>>> y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
>>> loss = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)
>>> assert loss.shape == (2,)
>>> loss.numpy()
array([0.0513, 2.303], dtype=float32)
Args:
y_true: Ground truth values.
y_pred: The predicted values.
from_logits: Whether `y_pred` is expected to be a logits tensor. By default,
we assume that `y_pred` encodes a probability distribution.
axis: Defaults to -1. The dimension along which the entropy is
computed.
Returns:
Sparse categorical crossentropy loss value.
def sparse_cat_loss(y_true,y_pred):
return sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)
def create_model(vocab_size, embed_dim, rnn_neurons, batch_size):
model = Sequential()
model.add(Embedding(vocab_size, embed_dim,batch_input_shape=[batch_size, None]))
model.add(GRU(rnn_neurons,return_sequences=True,stateful=True,recurrent_initializer='glorot_uniform'))
# Final Dense Layer to Predict
model.add(Dense(vocab_size))
model.compile(optimizer='adam', loss=sparse_cat_loss)
return model
model = create_model(
vocab_size = vocab_size,
embed_dim=embed_dim,
rnn_neurons=rnn_neurons,
batch_size=batch_size)
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) (128, None, 64) 5184
gru (GRU) (128, None, 1026) 3361176
dense (Dense) (128, None, 81) 83187
=================================================================
Total params: 3,449,547
Trainable params: 3,449,547
Non-trainable params: 0
_________________________________________________________________
Training the model¶
Let’s make sure everything is ok with our model before we spend too much time training! Let’s pass in a batch to confirm the model currently predicts random characters without any training.
for input_example_batch, target_example_batch in dataset.take(1):
# Predict off some random batch
example_batch_predictions = model(input_example_batch)
# Display the dimensions of the predictions
print(example_batch_predictions.shape, " <=== (batch_size, sequence_length, vocab_size)")
(128, 120, 81) <=== (batch_size, sequence_length, vocab_size)
# example_batch_predictions
sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)
# sampled_indices
# Reformat to not be a lists of lists
sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()
sampled_indices
array([66, 79, 76, 41, 5, 11, 74, 72, 70, 1, 73, 19, 26, 36, 30, 68, 0,
10, 78, 31, 73, 49, 13, 74, 38, 48, 76, 35, 51, 10, 54, 56, 22, 30,
63, 49, 65, 55, 61, 63, 31, 46, 76, 39, 58, 49, 68, 50, 50, 33, 45,
70, 27, 64, 47, 1, 18, 25, 23, 62, 21, 68, 8, 42, 60, 51, 11, 30,
48, 29, 18, 12, 43, 65, 24, 4, 17, 67, 32, 58, 30, 79, 21, 56, 35,
16, 42, 51, 23, 41, 38, 39, 70, 27, 72, 55, 37, 66, 20, 58, 24, 56,
57, 66, 65, 12, 45, 34, 26, 70, 4, 38, 43, 74, 25, 51, 69, 8, 75,
75], dtype=int64)
print("Given the input seq: \n")
print("".join(ind_to_char[input_example_batch[0]]))
print('\n')
print("Next Char Predictions: \n")
print("".join(ind_to_char[sampled_indices ]))
Given the input seq: of wonderful Kiss me while we both are still intangible [Chorus] Everybody needs salvation And if it doesn't come we m Next Char Predictions: n¡xP'.vtr u6AKEp -zFuY0vMWxJ[-bd9EkYmcikFUxNfYpZZHTrBlV 5?:j8p*Qh[.EWD5/Rm;&4oGfE¡8dJ3Q[:PMNrBtcLn7f;denm/TIAr&MRv?[q*ww
Alright, looks like everything’s working, we just need to train the network to learn from our small dataset, let’s train it!
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
Num GPUs Available: 1
epochs = 100
model.fit(dataset,epochs=epochs)
Generating text¶
Currently our model only expects 128 sequences at a time. We can create a new model that only expects a batch_size=1. We can create a new model with this batch size, then load our saved models weights. Then call .build() on the model:
model.save('weezer_gen.h5')
from tensorflow.keras.models import load_model
model = create_model(vocab_size, embed_dim, rnn_neurons, batch_size=1)
model.load_weights('weezer_gen.h5')
model.build(tf.TensorShape([1, None]))
model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_1 (Embedding) (1, None, 64) 5184
gru_1 (GRU) (1, None, 1026) 3361176
dense_1 (Dense) (1, None, 81) 83187
=================================================================
Total params: 3,449,547
Trainable params: 3,449,547
Non-trainable params: 0
_________________________________________________________________
def generate_text(model, start_seed,gen_size=100,temp=1.0):
'''
model: Trained Model to Generate Text
start_seed: Intial Seed text in string form
gen_size: Number of characters to generate
Basic idea behind this function is to take in some seed text, format it so
that it is in the correct shape for our network, then loop the sequence as
we keep adding our own predicted characters. Similar to our work in the RNN
time series problems.
'''
# Number of characters to generate
num_generate = gen_size
# Vecotrizing starting seed text
input_eval = [char_to_ind[s] for s in start_seed]
# Expand to match batch format shape
input_eval = tf.expand_dims(input_eval, 0)
# Empty list to hold resulting generated text
text_generated = []
# Temperature effects randomness in our resulting text
# The term is derived from entropy/thermodynamics.
# The temperature is used to effect probability of next characters.
# Higher probability == lesss surprising/ more expected
# Lower temperature == more surprising / less expected
temperature = temp
# Here batch size == 1
model.reset_states()
for i in range(num_generate):
# Generate Predictions
predictions = model(input_eval)
# Remove the batch shape dimension
predictions = tf.squeeze(predictions, 0)
# Use a cateogircal disitribution to select the next character
predictions = predictions / temperature
predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()
# Pass the predicted charracter for the next input
input_eval = tf.expand_dims([predicted_id], 0)
# Transform back to character letter
text_generated.append(ind_to_char[predicted_id])
return (start_seed + ''.join(text_generated))
print(generate_text(model,"Hey",gen_size=1000))
Hey, thirk you're wild When you're out there's gotta be more than this [Chorus] This is such a pity, we should give, eloo! [Chorus] Other way, other way I will turn and look the other way Other way, other way I will turn any nime anymoreahe that pean painful stars I hear them blahs harm meaning some om this I did what my body told me to I didn't mean to do youm ruse There's no better then, hen? [Chorus] I'm a troublemaker, never bight cliends Livifill our novep and every will remember prom night? Remember prom night Oo-oo-hoo-hoo [Verse 1] You wanna cry When you're crind [Verse 2] Keep on, blah, blah, blah We can go up (Hey), we go down (Hey) We gon' run you out of town We can go up (Hey), we go down) Yes, I'm down if you're down California snow, never let me got so gond In the mall, I was in the mall I was in the mall, I was in the mall I was in the mall Keep it steady I'm s your bat Give me feel, I can't tell you how the words hove and go Felland Hellont that thinks And I mate the g
Now you may not be very impressed by the results, but take a closer look at the output and remember that the model is predicting character by character! This means its actually starting to learn the structure of a song, you can see it begin to learn concepts of structure by utilizing whitespace and markers such as [Verse], [Bridge], and [Chorus] , which is incredible given how small the dataset is. We also see some evidence of overfitting as the model begins to just duplicate existing song lyrics (this is only something you can tell if you’re quite the Weezer fan and recognize the lyrics of multiple songs starting to merge).
Try taking this further by building out an even larger data set that includes more bands, or play around with the model hyperparameters or training epochs!
