An AI crawler is an automated program that visits public web pages for purposes such as search retrieval, model training, safety review, or data collection. Not every crawler has the same purpose, so a business should not assume that allowing one gives permission for every kind of AI use.
For example, OpenAI distinguishes OAI-SearchBot, which can access public pages for search-related uses, from GPTBot, which publishers may disallow for potential training use. A site’s robots.txt rules and any security layer in front of the site can affect whether these crawlers can reach a page.
Should every AI crawler be allowed?
That is a business decision, not a universal SEO setting. Decide separately what a crawler is for, whether the page is public, and whether technical controls are working as intended. If a business wants to be discoverable in a particular AI search product, it should verify that the relevant retrieval crawler is not blocked accidentally.