Teaching a Computer to Talk* *(Sort of)

Introduction

What’s the easiest way to teach a computer to talk?

This post shows a simple (interactive!) way to generate English text, providing an infinitely renewable source of Dr. Seuss stories and Doug Ford campaign speeches. The algorithm is available as an R package on GitHub, and there’s an interactive Shiny web app so you can try it yourself.

Motivation and research question

When it comes to computer-generated text, massive deep-learning systems like GPT-3 generate a lot of splashy headlines. But these systems are also opaque and have ridiculous environmental impacts–although it can be hard to measure them precisely. More to the point, they often don’t work in ways that can be hilarious, or, uh, not hilarious.

Reflecting on how poorly these large expensive models often work, I wondered how hard it would be to rig up a low-fi text generator using basic math. Or specifically:

• Can we generate semi-plausible English text modeled after an input text using simple and transparent methods?

The rest of this post will try to convince you that the answer is “yes, sort of,” by walking through the R package markovtext: A deeply unserious package for generating random text that mimics a given input text using Markov chains. Obviously this won’t have the generality or flexibility of GPT-3 and its friends, but I believe there’s value in seeing how far you can get with simple surveyable methods.

Skip the math, show me the interactive stuff!

Click here to play with the algorithm using an interactive Shiny web app.

It has a few default settings (Dr. Seuss, Doug Ford, Nietzsche, etc.) and you can also provide your own input text and adjust some of the parameters. You might try feeding it some classic literature, or perhaps something more contemporary.

Okay, now let’s talk math

What if we wanted to write English text probabilistically? We could start by choosing some words–say, 1000 of them–assigning each one a number, and then rolling a 1000-sided die and writing down the corresponding numbers. This would generate strings of English words, but you’re probably already thinking that it wouldn’t really be English text. We’d get all kinds of nonsense.

In coherent English words tend to follow specific other words. So we can’t just look at how often each specific word is used; we need to consider how often each specific word is used after one or more other words are used. We need a random system with some kind of a “memory.”

This is where we use Markov chains: very roughly, a Markov chain is a discrete stochastic process where the probability of future outcomes depends on the system’s present state.¹ While a coin flip famously “has no memory,”, a Markov chain does–just a little bit, at least.

This would let us ask the following question: given the word(s) we’ve just seen, which words are likely to come next? Both the words themselves and their likelihoods will depend on the input text we feed our algorithm.

So we put this all together into a simple model: We’ll generate text one word at a time using simple probabilities based on observed frequencies in an input text. We’ll break an input text down into a string of words (treating some punctuation marks as special words), count how often each word comes after each other word (or two words), and then use those counts to generate new text based on the words we’ve generated so far.

Generating some random text with R

You can install the development version of the R package markovtext from GitHub with:

# install.packages("devtools")
devtools::install_github("chris31415926535/markovtext")

The package has only two functions:

• markovtext::get_word_freqs() takes an input text and generates a word-frequency table.
• markovtext::generate_text() uses a word-frequency table to generate random text.

It also includes a few sample word-frequency tables to get you started. I’ll walk through some extremely legitimate use-cases here.

dougford_text <- markovtext::generate_text(markovtext::wordfreqs_dougford_3grams, word_length = 100)

knitr::kable(dougford_text, col.names = "")

seuss_text <- markovtext::generate_text(markovtext::wordfreqs_catinthehat_3grams, word_length = 100)

knitr::kable(seuss_text, col.names = "")

library(markovtext)

text <- "I can do wheelin', I can do dealin',
         But I don't do no damn squealin'.
         I can dig rappin', I'm ready! I can dig scrappin'.
         But I can't dig that backstabbin'."

wordfreqs <- markovtext::get_word_freqs(text, n_grams = 3)

new_text <- markovtext::generate_text(wordfreqs, word_length = 50)

knitr::kable(new_text, col.names = "")

Next steps?

I’m morbidly curious to see what would happen if you extend the “memory” back an arbitrary number of steps. My guess is that the output would become more grammatically correct, but that you’d also need to feed it much more data. In a short text, for example, any arbitrary string of 3 or 4 words is only likely to happen once, so your “random output” would just reproduce the input perfectly.

But now we’re getting beyond entertainment into more serious research, which is definitely outside the scope of this extremely unserious blog post.

Conclusion

You can generate semi-plausible English text using a surprisingly simple probabilistic model. The code to do so is available in a custom R package available on GitHub, and I’ve wrapped it into a nice interactive web app for users to explore themselves.

Did you try it out? Did it generate anything good?