You should not rely on robots when you want

[sumaiyakhatun26]

To hide pages with sensitive information. Even if you create a disallow directive for crawling in robots.txt, such a page can still be indexed by a search engine. Where to find the robot exclusion standard on your site The file is always located in the root of the site. The easiest way to find it is to search through all the site's files.


To do this, open the hosting admin panel and find the file manager. One of my list of benin whatsapp phone numbers sites is hosted on Beget , and there is a very nice file manager Sprutio. It allows you to search both by files and by text: The syntax of robots.txt is elementary: User-Agent: the name of the crawler. For example, Google's search robot is called Google-bot, and Yandex's crawler is simply Yandex.


Allow – allows scanning the page. Disallow – prevents the page from being scanned. Don't forget to add a link to the sitemap in your sitemap . For example, we want to show Google that all pages should be crawled except the contact page. The directive would be: User-Agent: Google-bot Allow: Disallow: / contacts Semantic core Let's talk about a few non-obvious points that will be useful for beginning optimizers: It is not necessary to buy paid tools for collecting the semantic core.