Friendly Robots

Friendly Robots makes adding a dynamic robots.txt file to your Umbraco website easy!

The package supports Umbraco 8.1+.

Usage

Once installed, the robots.txt file will be visible on the URL /robots.txt, such as https://www.yoursite.com/robots.txt.

If a physical robots.txt file exists in the root of the website, the dynamically generated file will be overwritten.

By default the robots.txt is automatically set to allow all traffic, but can be configured via a selection of app settings in the web.config.

Unless an alternative value is supplied, the "useragent" field will be catch-all ("*"). Multiple values can be supplied for each of the "allow", "disallow", and "sitemaps" URL fields as a comma separated list.