ChatGPT is one of the world’s largest and most powerful language models available to the public, and it feels like it can generate content on just about anything!
So, this can lead you to wonder – where does it get all of its information from? Does it learn from the people that use it and if so, how exactly does this work?
These are all excellent questions. In this article, I’ll be covering everything you need to know about how ChatGPT learns its information.
Does ChatGPT Learn From Users?
Although ChatGPT can temporarily retain information and context that you provide it, it does not actually change or improve itself based on the commands that you give it.
This means that the model is static, and doesn’t automatically improve over time.
In order for ChatGPT to have been created, it needed to be trained on a lot of data. Once the training was complete, it was released – and the model remains exactly the same, until OpenAI decides to train it some more.
Note that part of the reason OpenAI has released ChatGPT is for research purposes. They want to gain a better understanding of how the model will be used, and its shortcomings.
So, they allow users to submit feedback based on ChatGPT’s responses. This feedback (and the accompanying outputs) may be reviewed by OpenAI, and may possibly be used to help train the model further at a later date.
In fact, OpenAI’s FAQ for ChatGPT directly states that your conversations may be used to help improve their systems.
However, until OpenAI decides to train it again, the model is exactly the same – both for you, and for everyone else.
Note That Earlier Context Is Taken Into Account
If ChatGPT doesn’t learn from users, then how can it remember the information that you provided it?
Well, it turns out that language models like ChatGPT are able to retain a bit of context with the prompts that you send.
For example, ChatGPT can remember up to 4096 ‘tokens’ of context, with 1 token being equivalent to about 4 characters. As an estimate, this amounts to around 3,000 words.
So, if you’ve been using ChatGPT for multiple prompts, it doesn’t just take the prompt you just sent. It also takes up to around 3,000 words prior to that into account as well.
Note that even if ChatGPT appears to remember this context, this does not mean that changes have been permanently made to the model.
In fact, if someone else were to try and prompt the same thing, the context wouldn’t be there, meaning the output may be entirely different. And once the context is gone, it is gone forever.
In summary, ChatGPT may appear to be remember certain things you just told it, but this makes no changes to the underlying model itself. All of these ‘memories’ are erased as soon as you start a new prompt, or generate past the context window.
Note: Increases to the total context size, or unique ways to store key pieces of information for longer periods of time may be improvements that OpenAI introduces in the future. Their first public language model GPT-2 only had a context size of 1024 tokens, for example.
So, What Was ChatGPT Actually Trained On? Where Did It Learn From?
If ChatGPT didn’t learn from user inputs, where exactly did it get its information? How did it get so smart?
It turns out, ChatGPT was trained on large portions of the internet, an extremely large number of books, Wikipedia, and other large data sets that provide text.
One such example is Common Crawl, which you can learn more about on their website here.
In essence, it can be thought of similarly to if you read for 10,000 years straight, and stored every single word that you read… and that’s not an exaggeration!
ChatGPT’s predecessor, GPT-3 was trained on 45 terabytes of data, or over 500 billion words.
Because it’s been trained on so much of the entire internet, it is able to generate information on just about anything imaginable.
Even if it were to learn from user inputs, this would be such a tiny portion of its overall dataset that it wouldn’t make much of a difference for most topics.
Is ChatGPT Connected To The Internet?
One more question I’ve asked quite a bit – is ChatGPT connected to the internet at all?
Well, yes and no.
When you send a prompt to ChatGPT, this travels over the internet. OpenAI’s hardware processes the prompt, and returns the result back to you through the internet.
However, as mentioned earlier, ChatGPT does not update itself automatically. So, it is not constantly crawling the internet, looking for new data to add to itself.
Instead, this data must be assembled, and OpenAI would need to train ChatGPT further for it to learn from new information that’s been published.
In fact, the ‘PT’ in ‘GPT’ stands for pre-trained!
Anyhow, this is why ChatGPT may not be that familiar with recent events – at the time of writing, it was only trained up to 2019!
Currently, ChatGPT does not actually permanently learn anything from the data that you provide it.
While it may be able to temporarily retain data, this is tied to your current session. It does not actually improve the base model itself, so if you were to login under a new account or generate a new separate prompt, the information would be forgotten.
That being said, OpenAI is collecting feedback from users, and will likely use this as they continue to train the model further.
I hope that this article has answered some of your questions. If you have any other questions about ChatGPT, ask them below and I’ll be happy to help.
Wishing you the best,
– James McAllister