OpenAI, a renowned artificial intelligence lab, has recently revealed more information about its advanced web-crawler, GPTBot, which has been specifically engineered to peruse through websites and extract vital data. This extracted data then plays a pivotal role in enhancing the future AI models, such as ChatGPT 5 all future AI models, which heavily rely on it for better performance and functionality.
OpenAI explains that webpages crawled by GPTBot “may potentially be used to improve future models” and further advance new LLM models. By allowing GPTBot to sift through your website, you might be inadvertently contributing to the development of more accurate AI models that come with enhanced general capabilities and greater safety measures.
“Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII), or have text that violates our policies. Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety. Below, we also share how to disallow GPTBot from accessing your site.” – OpenAI
Sensitive data and paywall sites
Understanding the sensitivity and privacy concerns surrounding web data, OpenAI stresses that it has implemented stringent filters to safeguard against any breaches. Websites protected by paywalls or ones collecting personally identifiable information will stay beyond the reach of GPTBot. Moreover, any content that infringes on OpenAI’s policies will be off-limits for the web crawler.
However, this promising development does pose a complex issue. This newfound feasibility of potentially blocking OpenAI’s training scrapes, if acknowledged and respected, seems to have come a tad late to influence the current training data for models like ChatGPT or GPT-4. These models have already been trained with data that was stealthily scraped sans any prior announcements years ago.
For instance, OpenAI has collated data until September 2021, which symbolizes the current “knowledge” cutoff for its language models. Though these earlier models were trained without considerations for webpage consent, OpenAI’s new approach with GPTBot may indicate a shift towards increased transparency and respect for data source preferences in the future. With a more considerate and secure approach, AI’s future seems to be ensuring a balance between technological advancements and privacy preservation.
Stop ChatGPT from crawling your website
However, OpenAI understands that not all website owners may want their content to be accessible to GPTBot. Therefore, they have provided instructions on how to disallow GPTBot from accessing your site. This can be achieved by adding the GPTBot token to your site’s robots.txt file with the following lines:
User-agent: GPTBot
Disallow: /
For those who wish to customize GPTBot’s access, allowing it to access only specific parts of your site, you can modify your site’s robots.txt file as follows:
User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/
OpenAI’s crawler, including GPTBot, makes calls to websites from these specific IP address blocks. These currently include may change at a later date:
20.15.240.64/28
20.15.240.80/28
20.15.240.96/28
20.15.240.176/28
20.15.241.0/28
20.15.242.128/28
20.15.242.144/28
20.15.242.192/28
40.83.2.64/28
GPTBot privacy and data concerns
How to identify the OpenAI GPTBot in your logs and analytics
GPTBot is identifiable by its unique user agent token and full user-agent string. The user agent token is simply :
User agent token: GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
The primary function of GPTBot is to crawl web pages, which may then be used to refine and improve future AI models. It’s important to note that GPTBot is designed with privacy and security in mind. It filters out sources that require paywall access, gather personally identifiable information (PII), or contain text that violates OpenAI’s policies. By allowing GPTBot to access your site, you are contributing to the advancement of AI models, enhancing their accuracy, general capabilities, and safety.
GPTBot is a powerful tool in the AI landscape, designed to improve future models while respecting user privacy and content restrictions. Whether you choose to allow or disallow its access to your site, you now have the knowledge to make an informed decision.
For more details on how to stop the OpenAI ChatGPT Chat bot from accessing your website and scraping your content to create future AI models jump over to the official OpenAI website where more details are available.
Filed Under: Guides, Top News
Latest togetherbe Deals
Disclosure: Some of our articles include affiliate links. If you buy something through one of these links, togetherbe may earn an affiliate commission. Learn about our Disclosure Policy.
