You will then need to ensure that your particular robots.txt file is conserved. Make certain you perform some after making sure that internet crawlers should be able to find and recognize your file.
- Save the robots.txt file as being a text file.
- Put the file inside the directory that is highest-level of internet site ( or in the basis associated with domain).
- The file has got to be called robots.txt.
- Example: a robots.txt file that is conserved during the cause of test.com using the Address of http://www.sample.com/robots.txt is discoverable by internet crawlers, if the robots.txt file is positioned at a URL like http://www.sample.com/perhaps not_root/robots.txt it shall perhaps perhaps not manage to be located by internet crawlers.