Chapter 1: The Large Language Model Revolution

Imagine waking up to a beautiful, sunny morning. It’s Monday, and you know the week will be hectic. Your company is about to launch a new personal-productivity app, Taskr, and start a social media campaign to inform the world about your ingenious product.

This week, your main task is writing and publishing a series of engaging blog posts.

You start by making a to-do list:

You hit Enter, take a sip of coffee, and watch an article unravel on your screen, sentence by sentence, paragraph by paragraph. In 30 seconds, you have a meaningful, high-quality blog post, a perfect starter for your social media series. The visual is fun and attention-grabbing. It’s done! You choose the best title and begin the publishing process.

This is not a distant, futuristic fantasy but a glimpse of the new reality made possible by advancements in AI. As we write this book, many such applications are being created and deployed to a broader audience.

GPT-3 can solve general language-based tasks, like generating and classifying text, with unprecedented ease, moving freely between different text styles and purposes. The array of problems it can solve is vast.

In this book, we invite you to think of what problems you might solve with GPT-3 yourself. We’ll show you what it is and how to use it, but first, we want to give you a bit of context. The rest of this chapter will discuss where this technology comes from, how it is built, what tasks it excels at, and its potential risks. Let’s dive right in by looking at the field of natural language processing (NLP) and how large language models (LLMs) and GPT-3 fit into it.

A Behind-the-Scenes Look at Natural Language Processing

NLP is a subfield focusing on the interaction between computers and human languages. Its goal is to build systems that can process natural language, which is the way people communicate with each other. NLP combines linguistics, computer science, and artificial intelligence techniques to achieve this.

NLP combines the field of computational linguistics (rule-based modeling of human language) with machine learning to create intelligent machines capable of identifying the context and understanding the intent of natural language. Machine learning is a branch of AI that focuses on the study of how machines can improve their performance on tasks through experiences without being explicitly programmed to do so. Deep learning is a subfield of machine learning that involves using neural networks modeled after the human brain to perform complex tasks with minimal human intervention.

The 2010s saw the advent of deep learning, and with the maturity of the field came large language models consisting of dense neural networks composed of thousands or even millions of simple processing units called artificial neurons. Neural networks became the first significant game-changer in the field of NLP by making it feasible to perform complex natural language tasks, which had been possible only in theory so far. The second major milestone was the introduction of pre-trained models (such as GPT-3) that could be fine-tuned on various downstream tasks, saving many hours of training. (We discuss pre-trained models later in this chapter.)

NLP is at the core of many real-world AI applications, such as:

Language Models Getting Bigger & Better

The Generative Pre-Trained Transformer: GPT-3

The name GPT-3 stands for “Generative Pre-trained Transformer 3.” Let's go through all these terms one by one to understand the making of GPT-3.

Generative models

GPT-3 is a generative model because it generates text. Generative modeling is a branch of statistical modeling. It is a method for mathematically approximating the world.

Pre-trained models

A pre-trained model, keeping with Gladwell’s 10,000 hours theory, is the first skill you develop to help you acquire another faster. For example, mastering the craft of solving math problems can allow you to acquire the skill of solving engineering problems faster. A pre-trained model is trained (by you or someone else) for a more general task and can be fine-tuned for different tasks. Instead of creating a brand new model to address your issue, you can use a pre-trained model that has already been trained on a more general problem. The pre-trained model can be fine-tuned to address your specific needs by providing additional training with a tailored dataset. This approach is faster and more efficient and allows for improved performance compared to building a model from scratch.

Transformer Models

Neural networks are at the heart of deep learning, with their name and structure being inspired by the human brain. They are composed of a network or circuit of neurons that work together. Advances in neural networks can enhance the performance of AI models on various tasks, leading AI scientists to continually develop new architectures for these networks. One such advancement is the transformer, a machine learning model that processes a sequence of text all at once rather than one word at a time and has a strong ability to understand the relationship between those words. This invention has dramatically impacted the field of natural language processing.

Sequence-to-sequence models

The foundation of transformer models is sequence-to-sequence architecture. Sequence- to-sequence (Seq2Seq) models are useful for converting a sequence of elements, such as words in a sentence, into another sequence, such as a sentence in a different language. This is particularly effective in translation tasks, where a sequence of words in one language is translated into a sequence of words in another language. Google Translate started using a Seq2Seq-based model in 2016.

Transformer attention mechanisms

Transformer architecture was invented to improve AIs’ performance on machine translation tasks. “Transformers started as language models,” Kilcher explains, “not even that large, but then they became large."

To use transformer models effectively, it is crucial to grasp the concept of attention. Attention mechanisms mimic how the human brain focuses on specific parts of an input sequence, using probabilities to determine which parts of the sequence are most relevant at each step.

For example, look at the sentence,” The cat sat on the mat once it ate the mouse.” Does it in this sentence refer to “the cat" or "the mat"? The transformer model can strongly connect "it" with "the cat." That's attention.

An example of how the Encoder and Decoder work together is when the Encoder writes down important keywords related to the meaning of the sentence and provides them to the Decoder along with the translation. These keywords make it easier for the Decoder to understand the translation, as it now has a better understanding of the critical parts of the sentence and the terms that provide context.

Transformer models benefit from larger architectures and larger quantities of data. Training on large datasets and fine-tuning for specific tasks improve results. Transformers better understand the context of words in a sentence than any other kind of neural network. GPT is just the Decoder part of the transformer.

Now that you know what “GPT” means, let’s talk about that “3”- as well as 1 and 2.

GPT-3: A Brief History

GPT-1

GPT-1 was an essential stepping stone towards a language model with general language-based capabilities. It proved that language models can be effectively pre-trained, which could help them generalize well. The architecture could perform various NLP tasks with very little fine-tuning.

GPT-2

GPT-2 showed that training on a larger dataset and having more parameters improves a language model’s capability to understand tasks and surpass the state-of-the-art of many tasks in zero-shot settings. It also showed that even larger language models would better understand natural language.

To create an extensive, high-quality dataset, the authors scraped Reddit and pulled data from outbound links of upvoted articles on the platform. The resulting dataset, WebText, had 40GB of text data from over 8 million documents, far larger than GPT-1’s dataset. GPT-2 was trained on the WebText dataset and had 1.5 billion parameters, ten times more than GPT-1.

GPT-2 was evaluated on several datasets of downstream tasks like reading comprehension, summarisation, translation, and question answering.

GPT-3

In the quest to build an even more robust and powerful language model, OpenAI built the GPT-3 model. Both its dataset and the model are about two orders of magnitude larger than those used for GPT-2: GPT-3 has 175 billion parameters and was trained on a mix of five different text corpora, a much bigger dataset than the dataset used to train GPT-2. The architecture of GPT-3 is largely the same as GPT-2. It performs well on downstream NLP tasks in zero-shot and few-shot settings.

GPT-3 has capabilities like writing articles that are indistinguishable from human-written articles. It can also perform on-the-fly tasks for which it was never explicitly trained, like summing numbers, writing SQL queries, and even writing React and JavaScript codes given a plain English description of the tasks.

Note: Few, one, and zero-shot settings are specialized cases of zero-shot task transfer. In a few-shot setting, the model is provided with a task description and as many examples as fit into the context window of the model. The model is provided with exactly one example in a one-shot setting and in a zero-shot setting with no example.

OpenAI focuses on the democratic and ethical implications of AI in its mission statement. This can be seen in their decision to make the third version of their model, GPT-3, available via a public API. Application programming interface allows for a software intermediary to facilitate communication between a website or app and the user.

OpenAI researchers experimented with different model sizes while working on GPT-3. They took the existing GPT-2 architecture and increased the number of parameters. What came out as a result of that experiment was a model with new and extraordinary capabilities in the form of GPT-3. While GPT-2 displayed some zero-shot capabilities on downstream tasks, GPT-3  can carry out even more novel tasks when presented with example context.

Daniel Erickson, CEO of Viable, which has a GPT-3 powered product, praises the model’s ability to extract insights from large datasets through what he calls prompt-based development:

Companies going down that path cover use cases such as generating copy for ads and websites. The design philosophy is relatively simple: the company takes your data in, sends it over into a prompt, and displays the API-generated result. It solves a task that is easily done by a single API prompt and wraps a UI around that to deliver it to the users.

The problem Erickson sees with this category of use cases is that it is already overcrowded, attracting many ambitious startup founders competing with similar services. Instead, Erickson recommends looking at another use case, as Viable did. Data-driven use cases are not as crowded as prompt-generation use cases, but they are more profitable and allow you to create a moat easily.

The key, Erickson says, Erickson says, is to build a large dataset that you can keep adding to and that can provide potential insights. GPT-3 will help you extract valuable insights from it. At Viable, this was the model that let them monetize easily. “People pay much more for data than they do for prompt output,” Erickson explains.

It should be noted that technological revolutions also bring controversies and challenges. GPT-3 is a powerful tool in the hands of anyone trying to create a narrative. Without great care and benevolent intentions, one such challenge we will face is curbing the attempts to use the algorithm to spread misinformation campaigns. Another one would be eradicating its use for generating mass quantities of low-quality digital content that will then pollute the information available on the internet. Yet another one is the limitations of its datasets that are filled with various kinds of bias, which can be amplified by this technology. We will look closer at these and more challenges in Chapter 6, along with discussing the various efforts by OpenAI to address them.

Accessing the OpenAI API

As of 2021, the market has already produced several proprietary AI models with more parameters than GPT-3. However, access to them is limited to a handful of people within the company's R&D walls, making it impossible to evaluate their performance on real-world NLP tasks.

Another factor that makes GPT-3 accessible is its simple and intuitive “text-in, text-out” user interface. It doesn’t require complex gradient fine-tuning or updates, and you don’t need to be an expert to use it. This combination of scalable parameters and relatively open access makes GPT-3 the most exciting and arguably the most relevant language model to date.

Due to GPT-3's extraordinary capabilities, there are significant risks in terms of security and misuse associated with making it open-source, which we will cover in the last chapter—considering that, OpenAI decided not to publicly release the source code of GPT-3 and came up with a unique, never seen before access sharing model via an API.

The company initially decided to release API access in the form of a limited beta user list. There was an application process where people had to fill in a form detailing their background and the reasons for requesting API access. Only the approved users were granted access to a private beta of the API with an interface called Playground.

In its early days, the GPT-3 beta access waitlist consisted of tens of thousands of people. OpenAI swiftly managed the applications pouring in and adding developers in batches. It also closely monitored their activity and feedback about the API user experience to improve it continuously.

New users initially get a pool of free credits that allows them to freely experiment with the API. The number of credits is equivalent to creating text content as long as three average-length novels. After the free credits are used, users start paying for usage or, if they have a need, they can request additional credits from OpenAI API customer support.

Once you have signed up for an OpenAI account, you can move on to Chapter 2, where we will discuss the different components of the API, the GPT-3 Playground, and how to use the API to the best of its abilities for different use cases.