web_development:robots-txt

Robots.txt

https://www.inmotionhosting.com/support/website/how-to-stop-search-engines-from-crawling-your-website/ 

User-agent: *
Disallow: /

https://support.google.com/webmasters/answer/93710?hl=en 

<meta name="robots" content="noindex">
  • web_development/robots-txt.txt
  • Last modified: 2020/05/28 18:38
  • by jimboobrien