Time

If you just read the text and examples, this lesson is expected to take about 30 minutes to complete. In addition, there are reflection exercises, which are best done jointly with your fellow students. The video from Microsoft provides a deeper insight into the visions for the future use of large language models but is not necessary for understanding the technology behind.

About the lesson, 'The technology behind ChatGPT and large language models'

A large language model is an artificial intelligence which is optimized to understand and phrase human language. Therefore, we can talk or write to the language model in plain language, and it will respond to us by writing the texts we ask for, translating into other languages, answering our questions, summarizing information, and much more. But how do such models work? Learn the basics in this lesson.

Learning outcomes

Once you have completed the lesson, we expect you to be able to:

provide a basic explanation of what a large language model is, and how it works.
mention examples of common software products and online services that have long used AI language models, and where we can expect to encounter the technology in the future.
explain the difference between an AI language model and a search engine.
reflect on how AI language models can be used to streamline processes within your own field, and what pitfalls there might be in using the technology.

Source

The lesson was produced by the University of Copenhagen as part of its learning resources for digital literacy, 2024.

Course directors:

Anne Kirkegaard, Learning designer, Centre for Online and Blended Learning
Martin Rykov Ibsen, Learning designer, Centre for Online and Blended Learning
Andreas Lindenskov Tamborg, Tenure track assistant professor, Center for Digital Education, Department of Didactics in Science Education

____________________________________________________________

When ChatGPT came barging In

On November 30, 2022, the American company OpenAI launched a revolutionary chatbot, called ChatGPT. Suddenly, everyone with an Internet connection could play with an advanced AI language model, a machine that understands ordinary human language and can respond convincingly with intelligent texts in just a few seconds. Many actors, including researchers, open-source environments, and commercial companies, have been working intensely for years on the development of AI language models, but OpenAI was the first company to everyone access through a simple user interface on the Internet.

Universities and schoolteachers around the world panicked, journalists saw their jobs disappear, and programmers rejoiced at being able to solve their tasks in a fraction of the time they previously spent. After a couple of months, ChatGPT had broken all records for the fastest-growing number of users on an online service, ever. Microsoft invested heavily in OpenAI, and Google activated their crisis response system. This further accelerated the rapid development of AI language models, and the technology is therefore becoming increasingly integrated into many of the computer programs, apps, and online services we use in everyday life.

____________________________________________________________

AI language models before ChatGPT

Autocorrect in Word, auto-complete in text messages, Google Translate, and conversations with virtual assistants like Apple's Siri, Amazon's Alexa, and IKEA's unnamed chatbot – simple AI language models have been a part of our lives long before ChatGPT, and we have used the language models without giving much thought to the technology behind. However, the experience has not always been impressive, as the underlying language models were very small – and thus 'dumb' – compared to those we have access to in the wake of ChatGPT. A significant difference is that the earlier language models could only relate to a single sentence at a time and could not draw on the context from a longer conversation.

A simple AI language model can power, for example, auto-complete, Apple Messages, 2023.

____________________________________________________________

A vision for the future: Microsoft – The Future of Work with AI

On March 16, 2023, Microsoft delivered a fascinating and thought-provoking presentation of Microsoft 365 Copilot, where they revealed how they are working to integrate AI language models (also known as Large Language Models, LLMs) into the entire 365 suite of applications – and how each of us can use 'Copilot' as our personal assistant, since it can utilize data from all of our files, both in private, academic and work-related contexts.

Microsoft - the future of work with AI - 0615.png

Illustration from the video, 'The future of work with AI - Microsoft March 2023 Event', timecode 06:15

The video is 37 minutes long, and goes through many examples of how our lives become easier when we in plain language can ask Copilot to do the work for us. Do you need to do a PowerPoint presentation based on your bachelor's project? Click on the Copilot icon in PowerPoint, give your instructions in natural language, and seconds later you have a nice presentation based on the files from your bachelor project, and in the style of the slide presentations you have done before. The AI-generated presentation might not hit every mark, but you have a point of departure that you can work with.

It appears that AI language models will increasingly be integrated into the programs we use every day, and in this way, the boundaries between human and AI-generated content may become harder to differentiate. The development raises all kinds of questions that we won't delve into in this lesson, for example, about data ownership, copyright, plagiarism, source criticism, and the value of human versus synthetic content. See the lessons Digital Reflection and Digital Scientific Information Search for reflections on the consequences of AI language models becoming more widespread. In this lesson, the focus is on the technology itself.

If you are interested, you can watch the entire presentation 'The future of work with AI' here:

____________________________________________________________

The technology behind ChatGPT and other large language models

ChatGPT and other large language models are the result of a development process where machine learning and deep neural networks are used to create a statistical model that can understand and generate natural language (Natural Language Processing). This is complex technology, which combines many methods and is rapidly developing, so the following is a basic overview.

GPT stands for Generative Pre-trained Transformer:

Generative: The model generates new, unique content.

Pre-trained: The model is first 'pre-trained' on a large amount of data – in this case, texts from the Internet, including digitized books, Wikipedia, news sites, chat forums, and much more. During pre-training, the model uncovers patterns and relationships in the data without human supervision. Then, the model undergoes 'supervised fine-tuning', where humans guide the model to provide better responses and adhere to ethical frameworks.

Transformer: The model is a deep learning model of the transformer type. A deep learning model is a neural network that maps patterns and relationships in large amounts of data. When developing a language model, the model uncovers linguistic patterns by breaking the language down into smaller elements, also called 'tokens' (i.e., words, morphemes, characters, and spaces), and uncovering the statistical relationships between these tokens. This allows the model to understand the user's input and create text by predicting the most likely next character or word in a sentence. The transformer architecture, which was invented by engineers from Google in 2017, enables parallel processing of data instead of sequential, making the process much faster and more efficient. The transformer architecture allows the language model to process language at many levels at the same time – both as tokens, words, sentences, paragraphs, and full texts.

The language model – A gigantic web of 'tokens' and 'parameters'

DALL·E 2023-11-27 13.00.26 - Illustrate a futuristic and intricate spiderweb, representing Large Language Models.png

Section of an image created by DALL·E, November 27, 2023 - Prompt: 'a futuristic spiderweb representing a large language model'.

Imagine a large language model as a huge and complex web of linguistic elements (that is, words, morphemes, characters, and spaces – also called 'tokens') and the connections between them. Each token is like a node (artificial neuron) in the web, and each 'parameter' is like a thread connecting the nodes. Each parameter has a weight, which indicates the strength of the connection between nodes.

The number of parameters has exploded

The revolutionary development of AI language models has largely occurred by making the models larger and larger, partly thanks to the transformer architecture's efficient data processing. The more parameters a language model has, the more complex and detailed is the web – and the better it is at understanding and reproducing nuances in human language.

As the language models get more parameters they require more energy to train and use.

Below, you can see the development in the number of parameters in OpenAI's GPT language models from 2018 to 2023.

GPT-1, 2018	120.000.000
GPT-2, 2019	1.500.000.000
GPT-3, 2020	175.000.000.000
GPT-4, 2023	*) 1.700.000.000.000

*) Rumors say that GPT-4 may have up to 1.7 trillion parameters, but the numbers are not publicly available. GPT-4 is a multimodal model because it can understand and generate both text and images.

Large language models can be part of multimodal, generative AI systems

Generative AI can work with different modalities, i.e., different types of input and output:

Text (including computer code)
Images
Sound
Video

The earliest models would typically only take one type of input and generate one type of output, for example, text-to-text (GPT-3, Copilot) or text-to-image (Stable Diffusion, DALL-E, MidJourney). Newer models, such as GPT-4 and Google Gemini, are multimodal because they can take input in the form of text, sound, image, and video, and likewise generate output in these formats.

Sources:

Tokenization

Language can be converted into tokens in many different ways, and the chosen method is crucial for the quality of the model. For example, you can break down language into individual letters and characters (which offers high flexibility but requires a lot of computing power) or you can work with whole words (which offers low flexibility but requires less computing power). The large language models use 'subword tokenization', as this type of resolution of language elements provides good flexibility and at the same time saves on computing power. However, there is great variation in how the language models 'tokenize' language at the subword level. Below is an illustration of how language models of the type "distilbert" convert an input text into tokens and finally into numerical values that the language model can work with.

Demystifying tokenization – subword tokenization.png

Illustration from: Demystifying Tokenization: The Building Blocks of Language AI

How does the language model come up with its responses?

When you prompt a language model, the model processes the input by breaking it up into tokens, but also by analyzing words, sentences, and text in a larger context. The language model uses its large statistical model to predict the next token in its response. The token with the highest probability is usually – but not always – chosen as the next in the response, and the process is repeated until the language model reaches a 'stop criterion', which is set at a maximum number of tokens. While the model generates its response, it draws on both the context of the input text and the output it has managed to produce. This process, where the model continuously runs iterations over the prediction process, is called 'autoregression'.

Language models are 'non-deterministic' – meaning there is variation in the output, even if you give the same input. Therefore, you can ask the model to generate another response from the same input – and the result will be a new, unique response. This happens because the model does not always select the most likely next token in its response but has a built-in parameter that determines how much it can deviate. This parameter is called 'temperature'. If the temperature is set to 0, the model will always choose the most likely next token, but if the temperature is >0, the model will be more 'creative' in its responses, as it occasionally chooses a different token than the most likely one.

Demystifying temperature.png

Illustration from: Demystifying the Temperature Parameter: A Visual Guide to Understanding its Role in Large Language Models

______________________________________________________

Search engines and AI language models – a good match or a dangerous cocktail?

Language models generate new, unique texts that draw on the vast text material the model has been trained on – and OpenAI's GPT models are optimized to be friendly and convincing conversation partners. A search engine, on the other hand, searches in existing content that is available on the Internet. The search engine uses techniques such as indexing, crawling, ranking, and matching to find and sort the most relevant results for the user's query.

The technologies behind search engines and large language models can be combined, and below you can see some examples of how they work separately and in concert.

A language model has learned from the Internet’s texts – but can function offline

When training a large language model the developers use a fixed set of data which does not change. The version of ChatGPT that is freely available in January 2024 runs on OpenAI's language model, GPT-3.5. The model is primarily trained on text material published up to September 2021, and since then it has been updated with 'knowledge' up until January 2022. Therefore, ChatGPT does not know about events thereafter.

If you – in January 2024 – ask about Russia's invasion of Ukraine you get the following answer:
(click on the question to read ChatGPT's answer)

What happened when Russia invaded Ukraine?

ChatGPT - answer icon.png ChatGPT responded (January 2, 2024):

As of my last knowledge update in January 2022, I cannot provide real-time information on events that may have occurred after that date. Up until that point, Russia's invasion of Ukraine had not taken place. The situation in international affairs is dynamic, and events may have unfolded since then.

To get the most recent and accurate information about the situation, I recommend checking the latest news updates from reliable sources or official government statements. News agencies, government websites, and international organizations can provide up-to-date information on geopolitical events.

Although you accesses ChatGPT via the Internet, the language model (GPT-3.5) does not search the Internet when generating its responses – it only draws on knowledge from the dataset it has been trained on.

New language models can read updated information on the Internet

Microsoft, which has entered into a partnership with OpenAI, integrated OpenAI's latest language model, GPT-4, into Copilot in 2023 – a new chat feature that is available alongside Microsoft's search engine, Bing. GPT-4 is trained on data collected up until April 2023, and can therefore provide information about the Russian invasion of Ukraine in 2022. Copilot can also read updated information on the Internet and provide sources for the information it gives.

(click on the question to read Copilot's answer)

What happened when Russia invaded Ukraine?

Bing chat logo.png Microsoft Copilot responded (January2, 2024):

The Russian invasion of Ukraine began in February 2014 with the covert invasion of the Ukrainian autonomous republic of Crimea by disguised Russian troops. The conflict expanded in April 2014 when Russians and local proxy forces seized territory in Ukraine’s Donbas region. Over the next seven years, more than 14,000 people would be killed in fighting in eastern Ukraine¹. On February 24, 2022, Russia launched a full-scale invasion of Ukraine. Although Russian forces made significant gains in the first days of combat, Ukrainian defenders rebuffed attempts to seize Kyiv and other major cities and were soon launching counterattacks at Russian positions¹. The war is still ongoing and has caused significant damage to both countries¹.

I hope this helps. Let me know if you have any other questions.

Source(s)

Hallucinations – sometimes language models create nonsense

Occasionally, language models invent information that does not appear in the training data. This is called 'hallucinations'. Developers work intensively to minimize the problem. The risk of hallucinations increases in areas where data is limited – for example, if you ask about very current events.

In her New Year's speech on December 31, 2023, Queen Margrethe announced that she would step down as queen on January 14, 2024. On January 2, 2024 – that is, two days after the announcement, but 12 days before the abdication – Copilot responded as follows:

(click on the question to read Copilot's answer)

When will the Danish queen resign?

BingChat When will the queen resign.png

In this instance, the errors might occur because the language model has not yet 'read' that many texts describing the event – and therefore the statistical material is not sufficient to ensure a coherent answer with the correct date.

If you, on the other hand, use Bing's search engine, you get key quotes taken directly from updated sources, correctly reporting the facts, and you also get the source links:

(click on the question to read the result from Bing Search, January 2, 2024)

BingSearch Result Queens resignation.png

______________________________________________________

Reflection exercise

Here is inspiration for testing and assessing how AI language models work. Use one or more language models you can access, such as ChatGPT (OpenAI), Copilot or Gemini (Google).

Please note! Staff and students at UCPH have free access to Microsoft Copilot. Remember that you must login to Microsoft using your university credentials to get access to the latest version of Copilot and to ensure that your data are protected by the university's data management agreement with Microsoft.

Try to create an original piece of content. It could be a short story, song lyrics, or a poem. Then analyze the result to assess ability of the model to mimic a creative process. Try several language models if possible. Consider questions like:

In what ways did the model succeed in being creative, and where did it fail?
What influence do you think AI will have on creative professions?

Design an experiment where you test the ability of a large language model to identify true vs. false information. Ask the language model to verify a series of statements (where you know the answer) and assess its responses. Then discuss how the model handles fact-checking and what limitations it may have. How can these observations influence trust in AI-generated answers?

How can AI language models shape our labor market, education system, and daily life? What impact do you think AI language models will have on your specific field? Think about positive and negative aspects, such as increased productivity and automation of tasks, versus the risk of errors and job loss.
What strategies can we employ to maximize the positive effects while minimizing the negative?

____________________________________________________________

If you want to know more

Here you will find a number of resources where you can independently learn more about large language models and overall generative AI:

Generative AI exists because of the transformer – This is how it works

Financial Times has created a thorough visual explanation of how the large language models work.

Intro to Large Language Models - YouTube

Andrej Karpathy explains the technological background of LLMs. He is a Slovak-Canadian computer scientist who has worked as the Director of Artificial Intelligence at Tesla and is now employed at OpenAI, where he among other things specializes in deep learning.

Prompt | DR LYD (only in Danish)

The future is here: Artificial intelligence can do everything from diagnosing you, composing the perfect pop song – and maybe even wage a war. Every week, tech analyst Marcel Mirzaei-Fard and his close colleague Henrik Moltke, DR's technology correspondent, closely read the biggest stories about the technological development that is changing everything we know – and ask whether AI is a friend or foe.

The New York Times - Hard Fork

Hard Fork is a podcast from The New York Times about a future that is already here. Every week, journalists Kevin Roose and Casey Newton explore and try to make sense of the latest in the rapidly changing world of technology. We recommend that you start with the episode 'Generative AI is here – Who should control it?', published October 21, 2022.

Dagbladet Information - Maskinstorm (only in Danish)

How will artificial intelligence change our world? In the podcast 'Maskinstorm', a number of Information's specialist staff invite experts into the studio to try to and find answers to that very question.

Additional sources::

____________________________________________________________