Create a robots.txt file to control which pages search engine crawlers can access.
2 free uses remaining. Sign up free for unlimited.
About
The robots.txt file tells search engine bots which pages to crawl or skip. Place it at your domain root. Use presets to quickly configure common setups.