Introduction

Language models and AI chatbots have become increasingly prevalent in our digital interactions. They assist us in various tasks, but have you ever wondered where they acquire their vast knowledge? In this article, we will explore the sources of information for language models, debunk some common misconceptions, and discuss the need for curated data to enhance their capabilities.

Unveiling the Data Sources

Many people mistakenly believe that language models like GPT (Generative Pre-trained Transformer) are trained on the entirety of the internet. However, this notion is far from accurate. In reality, language models are trained on a relatively small amount of data. The pre-trained GPT models are not trained on the internet itself but rather on a limited dataset, which can fit on a laptop computer. The claim that GPT is trained on the entire internet is pure misinformation.

Data Quantity vs. Data Quality

The limited training data used for language models is intentional and serves a purpose. If a language model were trained on a vast amount of data, including the entire internet, the data quality would suffer. As the saying goes, "garbage in, garbage out." Poor-quality training data would yield inaccurate and unreliable results. Therefore, it is crucial to ensure that language models have a solid foundation of high-quality data to build upon.

Understanding the Scale

To grasp the scale of the data used to train language models, let's consider some analogies. If we compare the training data of GPT to a swimming pool, the internet would be akin to all the world's oceans, lakes, and ponds combined. Another analogy is that if the GPT training data were the size of a 2,000 square foot home, the internet would be an entire city with countless homes. These examples highlight the vast difference in scale between the training data and the internet.

Data Sources and Limitations

While language models like GPT utilize various data sources, one prominent contributor is Wikipedia. However, it's important to note that Wikipedia is edited by a relatively small group of individuals. Additionally, language models are trained on a corpus of books, but not every single book can be included due to copyright restrictions. Therefore, the training data is not all-encompassing, and there are limitations to the sources from which language models derive their information.

The Need for Curated Data

To improve language models, it is crucial to expand the dataset and ensure its curation by human experts. The AI's performance won't progress significantly without more diverse and carefully selected training data. As we advance beyond relying on limited filtered data from sources like Wikipedia, specific datasets will be necessary. For instance, it would be invaluable to have the ability to limit responses to particular books or sets of books. This targeted approach would provide deeper insights and accuracy, akin to looking inside a matchbox versus searching an entire house.